Oct 30 13:02:28.439766 ip-10-0-136-36 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 13:02:28.839334 ip-10-0-136-36 kubenswrapper[2574]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:02:28.839334 ip-10-0-136-36 kubenswrapper[2574]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 13:02:28.839334 ip-10-0-136-36 kubenswrapper[2574]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:02:28.839334 ip-10-0-136-36 kubenswrapper[2574]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:02:28.844790 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844701    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.844790 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844786    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.844790 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844790    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.844790 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844794    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.844790 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844800    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844804    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844809    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844812    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844817    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844821    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844824    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844828    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844832    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844836    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844840    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844844    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844847    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844852    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844857    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844860    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844864    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844869    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844873    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844877    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.845090 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844881    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844885    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844889    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844893    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844897    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844901    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844905    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844909    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844913    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844917    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844921    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844926    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844930    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844934    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844939    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844946    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844953    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844958    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844963    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.846005 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844968    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844974    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844979    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844984    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844988    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844992    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.844997    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845001    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845005    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845009    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845015    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845019    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845024    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845028    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845032    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845036    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845040    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845045    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845050    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845054    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.846806 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845058    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845063    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845067    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845071    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845075    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845079    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845091    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845097    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845101    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845107    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845112    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845117    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845121    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845126    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845130    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845135    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845139    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845143    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845147    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.847646 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845152    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845156    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845161    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845165    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845172    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845177    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845182    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845186    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845190    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845194    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845199    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845203    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845208    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845211    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845215    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845219    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845223    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845227    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845231    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.848517 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845235    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845240    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845244    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845248    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.845253    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846672    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846682    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846687    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846693    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846697    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846702    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846706    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846711    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846715    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846720    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846724    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846728    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846732    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846737    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846742    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.849326 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846746    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846751    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846755    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846759    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846764    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846769    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846774    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846779    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846783    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846787    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846792    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846796    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846801    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846807    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846813    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846818    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846822    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846828    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846832    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.850198 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846837    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846841    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846846    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846850    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846854    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846858    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846862    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846867    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846874    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846881    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846886    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846890    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846895    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846899    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846903    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846908    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846912    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846917    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846922    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.851041 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846927    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846932    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846936    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846940    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846945    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846949    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846953    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846957    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846961    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846965    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846970    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846974    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846979    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846983    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846988    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846992    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.846996    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847000    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847005    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847009    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.851605 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847013    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847018    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847023    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847027    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847032    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847036    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847041    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847045    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847049    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847054    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847059    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847064    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847068    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847072    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847077    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847081    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847085    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847089    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847094    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847098    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.852310 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847102    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847106    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847111    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847115    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847119    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847123    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847128    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847132    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847136    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847140    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847145    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847149    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.847155    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848078    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848089    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848095    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848099    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848104    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848108    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848113    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.852825 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848118    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848122    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848127    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848132    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848136    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848141    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848145    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848149    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848154    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848158    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848162    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848166    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848170    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848175    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848179    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848183    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848187    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848201    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848209    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848213    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.853397 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848218    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848222    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848226    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848229    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848234    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848238    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848242    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848246    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848251    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848255    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848259    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848264    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848268    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848271    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848276    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848280    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848284    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848288    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848292    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.853904 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848296    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848301    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848305    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848309    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848323    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848329    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848333    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848338    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848342    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848346    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848351    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848370    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848375    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848379    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848384    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848388    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848392    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848396    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848401    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.854365 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848407    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848413    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848419    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848424    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848429    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848434    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848439    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848443    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848447    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848451    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848455    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848459    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848464    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848468    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848473    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848477    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848481    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848485    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848505    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848509    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.854853 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848513    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848518    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848522    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848525    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848531    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848543    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848547    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848551    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848556    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848560    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848564    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848568    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848572    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848578    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848582    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848587    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848591    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848595    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848600    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.855317 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848605    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.848609    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848747    2574 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848758    2574 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848771    2574 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848778    2574 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848785    2574 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848790    2574 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848796    2574 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848803    2574 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848808    2574 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848813    2574 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848818    2574 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848823    2574 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848828    2574 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848833    2574 flags.go:64] FLAG: --cgroup-root=""
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848838    2574 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848843    2574 flags.go:64] FLAG: --client-ca-file=""
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848848    2574 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848860    2574 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848871    2574 flags.go:64] FLAG: --cluster-domain=""
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848876    2574 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848908    2574 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848915    2574 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 13:02:28.855779 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848921    2574 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848928    2574 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848934    2574 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848939    2574 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848944    2574 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848949    2574 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848954    2574 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848959    2574 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848964    2574 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848970    2574 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848977    2574 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848982    2574 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848987    2574 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848992    2574 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.848996    2574 flags.go:64] FLAG: --enable-server="true"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849001    2574 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849012    2574 flags.go:64] FLAG: --event-burst="100"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849017    2574 flags.go:64] FLAG: --event-qps="50"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849022    2574 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849026    2574 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849031    2574 flags.go:64] FLAG: --eviction-hard=""
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849038    2574 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849043    2574 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849048    2574 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849053    2574 flags.go:64] FLAG: --eviction-soft=""
Oct 30 13:02:28.856351 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849058    2574 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849063    2574 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849068    2574 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849090    2574 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849096    2574 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849101    2574 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849106    2574 flags.go:64] FLAG: --feature-gates=""
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849113    2574 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849118    2574 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849123    2574 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849128    2574 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849133    2574 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849138    2574 flags.go:64] FLAG: --help="false"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849143    2574 flags.go:64] FLAG: --hostname-override="ip-10-0-136-36.ec2.internal"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849148    2574 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849153    2574 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849158    2574 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849163    2574 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849169    2574 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849174    2574 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849178    2574 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849183    2574 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849188    2574 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 13:02:28.857040 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849192    2574 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849198    2574 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849202    2574 flags.go:64] FLAG: --kube-reserved=""
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849207    2574 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849212    2574 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849217    2574 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849222    2574 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849226    2574 flags.go:64] FLAG: --lock-file=""
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849231    2574 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849235    2574 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849240    2574 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849249    2574 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849254    2574 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849267    2574 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849272    2574 flags.go:64] FLAG: --logging-format="text"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849276    2574 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849282    2574 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849286    2574 flags.go:64] FLAG: --manifest-url=""
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849291    2574 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849298    2574 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849308    2574 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849314    2574 flags.go:64] FLAG: --max-pods="110"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849319    2574 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849324    2574 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849328    2574 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 13:02:28.857615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849333    2574 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849338    2574 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849343    2574 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849347    2574 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849362    2574 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849368    2574 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849373    2574 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849378    2574 flags.go:64] FLAG: --pod-cidr=""
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849382    2574 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849387    2574 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849392    2574 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849396    2574 flags.go:64] FLAG: --port="10250"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849401    2574 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849406    2574 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cdb13b08d46689ee"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849411    2574 flags.go:64] FLAG: --qos-reserved=""
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849416    2574 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849421    2574 flags.go:64] FLAG: --register-node="true"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849426    2574 flags.go:64] FLAG: --register-with-taints=""
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849432    2574 flags.go:64] FLAG: --registry-burst="10"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849437    2574 flags.go:64] FLAG: --registry-qps="5"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849441    2574 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849455    2574 flags.go:64] FLAG: --reserved-memory=""
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849462    2574 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849467    2574 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849472    2574 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849476    2574 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 13:02:28.858231 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849481    2574 flags.go:64] FLAG: --runonce="false"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849503    2574 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849509    2574 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849514    2574 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849519    2574 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849523    2574 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849528    2574 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849533    2574 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849539    2574 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849544    2574 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849549    2574 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849553    2574 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849558    2574 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849563    2574 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849568    2574 flags.go:64] FLAG: --system-cgroups=""
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849572    2574 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849582    2574 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849586    2574 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849591    2574 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849602    2574 flags.go:64] FLAG: --tls-min-version=""
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849606    2574 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849611    2574 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849618    2574 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849623    2574 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849629    2574 flags.go:64] FLAG: --v="2"
Oct 30 13:02:28.858857 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849636    2574 flags.go:64] FLAG: --version="false"
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849642    2574 flags.go:64] FLAG: --vmodule=""
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849649    2574 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.849662    2574 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849941    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849948    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849953    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849957    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849963    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849968    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849973    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849977    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849981    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849986    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849990    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.849995    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850000    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850004    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850009    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850013    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.859452 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850017    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850022    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850026    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850030    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850034    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850038    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850043    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850047    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850051    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850057    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850061    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850065    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850069    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850074    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850078    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850082    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850087    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850092    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850096    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850100    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.859945 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850106    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850110    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850114    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850119    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850123    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850128    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850132    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850137    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850141    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850145    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850149    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850153    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850157    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850162    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850166    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850170    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850175    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850179    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850186    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.860416 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850191    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850196    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850203    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850206    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850210    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850215    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850218    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850223    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850227    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850231    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850237    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850241    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850246    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850252    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850256    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850260    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850264    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850269    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850274    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.860948 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850278    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850283    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850287    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850292    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850296    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850300    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850307    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850313    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850318    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850322    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850327    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850331    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850335    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850339    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850343    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850349    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850353    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850358    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850362    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.861414 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850367    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850371    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850375    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850379    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850384    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850388    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850393    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850399    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850404    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850408    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850413    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850417    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850421    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.850463    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850540    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850549    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.861873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850555    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850560    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850564    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850569    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850573    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850577    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850582    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850586    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850590    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850594    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850599    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850603    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850609    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850613    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850617    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850622    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850626    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850630    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850635    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850639    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.862258 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850643    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850647    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850653    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850657    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850663    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850668    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850672    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850677    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850681    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850685    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850690    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850694    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850698    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850702    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850707    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850711    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850715    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850719    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850724    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850728    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.862729 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850733    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850739    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850745    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850751    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850757    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850762    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850766    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850771    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850775    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850779    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850783    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850787    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850792    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850796    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850801    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850806    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850812    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850816    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850820    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.863195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850827    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850831    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850836    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850840    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850845    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850849    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850853    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850858    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850864    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850871    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850877    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850884    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850890    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850894    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850899    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850903    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850907    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850913    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850917    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.863677 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850921    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850926    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850930    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850934    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850938    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850943    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850947    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850951    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850955    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850959    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850966    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850971    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850975    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850980    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850984    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850988    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850992    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.850996    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851000    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851004    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.864114 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851009    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851013    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851018    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851022    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851028    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.851032    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.858117    2574 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.858134    2574 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.858150    2574 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.858154    2574 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858209    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858214    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858218    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858221    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858224    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858227    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858229    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858233    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858236    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858239    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.864695 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858242    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858245    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858247    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858250    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858253    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858256    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858258    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858261    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858264    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858266    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858269    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858271    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858274    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858276    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858279    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858282    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858284    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858287    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858290    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858294    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.865462 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858297    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858299    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858302    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858305    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858308    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858310    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858313    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858317    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858320    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858324    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858326    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858329    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858332    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858334    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858337    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858340    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858342    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858345    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858347    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858349    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.866255 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858352    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858355    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858357    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858360    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858362    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858365    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858368    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858370    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858373    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858376    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858378    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858381    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858385    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858387    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858390    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858393    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858395    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858398    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858401    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858403    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.866840 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858406    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858408    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858411    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858414    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858416    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858419    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858422    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858424    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858427    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858429    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858432    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858434    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858438    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858440    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858443    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858446    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858448    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858452    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858456    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.867339 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858459    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858462    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858465    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858468    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858471    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858473    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858477    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858480    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858482    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858485    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858499    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858502    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858504    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858507    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858510    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858512    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.867795 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858515    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.858543    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858714    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858718    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858721    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858724    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858727    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858729    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858732    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858734    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858737    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858740    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858742    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858745    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858747    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858750    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.868223 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858752    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858755    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858759    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858762    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858765    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858768    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858770    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858774    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858776    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858779    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858781    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858785    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858789    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858793    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858796    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858799    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858802    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858805    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858808    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.868744 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858811    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858814    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858816    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858819    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858821    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858824    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858826    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858829    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858832    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858834    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858837    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858840    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858843    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858845    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858848    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858850    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858853    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858855    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858858    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.869219 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858860    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858863    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858866    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858869    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858872    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858875    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858877    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858880    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858882    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858885    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858887    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858890    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858892    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858895    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858897    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858900    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858902    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858904    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858907    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858910    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.869684 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858912    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858915    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858918    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858920    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858923    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858925    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858927    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858930    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858933    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858935    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858937    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858940    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858942    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858945    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858947    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858950    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858953    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858956    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858959    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858961    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.870195 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858964    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858966    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858969    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858971    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858974    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858976    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858979    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858981    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858984    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858986    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858989    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858991    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858993    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.858996    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.859018    2574 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:02:28.870766 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859049    2574 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859053    2574 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859056    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859058    2574 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859061    2574 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859063    2574 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859066    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859068    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859071    2574 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859073    2574 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859076    2574 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859078    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859081    2574 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859084    2574 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859086    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859089    2574 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859091    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859094    2574 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859096    2574 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:02:28.871137 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859100    2574 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859103    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859106    2574 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859108    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859111    2574 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859114    2574 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859116    2574 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859119    2574 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859121    2574 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859125    2574 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859129    2574 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859132    2574 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859134    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859137    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859139    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859142    2574 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859145    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859147    2574 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859150    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:02:28.871665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859152    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859154    2574 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859157    2574 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859159    2574 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859162    2574 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859164    2574 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859167    2574 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859169    2574 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859172    2574 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859175    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859178    2574 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859180    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859183    2574 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859185    2574 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859187    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859190    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859192    2574 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859195    2574 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859198    2574 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859200    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:02:28.872103 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859202    2574 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859207    2574 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859210    2574 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859213    2574 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859215    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859218    2574 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859221    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859224    2574 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859226    2574 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859229    2574 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859231    2574 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859234    2574 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859237    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859239    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859241    2574 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859244    2574 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859247    2574 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859249    2574 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859252    2574 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859255    2574 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:02:28.872623 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859257    2574 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859260    2574 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859263    2574 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859266    2574 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859268    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859271    2574 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859273    2574 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859276    2574 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859278    2574 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859281    2574 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859283    2574 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859286    2574 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859288    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859292    2574 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859295    2574 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859297    2574 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859300    2574 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859302    2574 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859305    2574 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859308    2574 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:02:28.873105 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859310    2574 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859313    2574 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859316    2574 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859318    2574 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859320    2574 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859323    2574 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859325    2574 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:28.859329    2574 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.859380    2574 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.861322    2574 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.862229    2574 server.go:1014] "Starting client certificate rotation"
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.862323    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.862365    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.866687    2574 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:02:28.873581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.871287    2574 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:02:28.882621 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.882604    2574 log.go:25] "Validated CRI v1 runtime API"
Oct 30 13:02:28.885790 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.885775    2574 log.go:25] "Validated CRI v1 image API"
Oct 30 13:02:28.886678 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.886647    2574 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 13:02:28.891575 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.891546    2574 fs.go:135] Filesystem UUIDs: map[75903a49-d32c-4be3-8bde-60ac9c3f35c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9fb4bca6-2e7a-4eb1-9a77-ce5e69419e63:/dev/nvme0n1p3]
Oct 30 13:02:28.891630 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.891570    2574 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 13:02:28.897578 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.897455    2574 manager.go:217] Machine: {Timestamp:2026-10-30 13:02:28.896148823 +0000 UTC m=+0.330147738 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098910 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20e04913e21e000a00bfbb5e67092a SystemUUID:ec20e049-13e2-1e00-0a00-bfbb5e67092a BootID:fad49515-fd18-475a-905c-6cf100944b63 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:de:d0:d7:6f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:de:d0:d7:6f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:9d:9c:bf:e6:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 13:02:28.898146 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.898136    2574 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 13:02:28.898234 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.898223    2574 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 13:02:28.900903 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.900878    2574 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 13:02:28.901036 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.900904    2574 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-36.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 13:02:28.901092 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.901041    2574 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 13:02:28.901092 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.901049    2574 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 13:02:28.901092 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.901079    2574 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:02:28.901172 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.901100    2574 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:02:28.901849 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.901839    2574 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 13:02:28.905365 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.905353    2574 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 13:02:28.905484 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.905475    2574 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 13:02:28.906210 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.906201    2574 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 13:02:28.906247 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.906215    2574 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 13:02:28.906247 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.906226    2574 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 13:02:28.906247 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.906235    2574 kubelet.go:405] "Adding apiserver pod source"
Oct 30 13:02:28.906247 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.906244    2574 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 13:02:28.906601 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.906586    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:02:28.909972 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.909956    2574 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 13:02:28.914722 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.914707    2574 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:02:28.915599 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.915585    2574 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 13:02:28.915666 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.915625    2574 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 13:02:28.916670 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.916657    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 13:02:28.916741 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.916678    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 13:02:28.916741 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.916687    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 13:02:28.916741 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.916696    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 13:02:28.916741 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.916704    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 13:02:28.916741 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.916715    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 13:02:28.917192 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917182    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 13:02:28.917244 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917194    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 13:02:28.917244 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917207    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 13:02:28.917244 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917216    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 13:02:28.917244 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917237    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 13:02:28.917386 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917251    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 13:02:28.917867 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917857    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 13:02:28.917916 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.917870    2574 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 13:02:28.921551 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.921537    2574 server.go:1263] "Started kubelet"
Oct 30 13:02:28.922232 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.922162    2574 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 13:02:28.922345 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.922225    2574 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 13:02:28.922391 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.922374    2574 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 13:02:28.923860 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.923772    2574 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 13:02:28.924680 ip-10-0-136-36 systemd[1]: Started Kubernetes Kubelet.
Oct 30 13:02:28.926728 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.926710    2574 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 13:02:28.931330 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.931311    2574 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 13:02:28.931411 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.931337    2574 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 13:02:28.931411 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.931376    2574 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 13:02:28.931537 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.931423    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:28.931912 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.931892    2574 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 13:02:28.931912 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.931893    2574 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 13:02:28.931912 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.931916    2574 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 13:02:28.932108 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932002    2574 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 13:02:28.932108 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932010    2574 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 13:02:28.932270 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932255    2574 factory.go:55] Registering systemd factory
Oct 30 13:02:28.932270 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932272    2574 factory.go:223] Registration of the systemd container factory successfully
Oct 30 13:02:28.932536 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932522    2574 factory.go:153] Registering CRI-O factory
Oct 30 13:02:28.932600 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932540    2574 factory.go:223] Registration of the crio container factory successfully
Oct 30 13:02:28.932600 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932593    2574 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 13:02:28.932679 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932611    2574 factory.go:103] Registering Raw factory
Oct 30 13:02:28.932679 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.932620    2574 manager.go:1199] Started watching for new ooms in manager
Oct 30 13:02:28.933537 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.933523    2574 manager.go:319] Starting recovery of all containers
Oct 30 13:02:28.942009 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.941975    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 13:02:28.943996 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.943978    2574 manager.go:324] Recovery completed
Oct 30 13:02:28.949071 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.949055    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:28.951128 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.951105    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:28.951203 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.951155    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:28.951260 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.951202    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:28.952257 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.952241    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:28.952339 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.952248    2574 cpu_manager.go:226] "Starting" policy="none"
Oct 30 13:02:28.952339 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.952305    2574 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 13:02:28.952339 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.952328    2574 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 13:02:28.952936 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.952920    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:28.953010 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.952944    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:28.953010 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.952957    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:28.953214 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.953187    2574 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 13:02:28.953290 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.953185    2574 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-36.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 13:02:28.954137 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.953192    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-36.ec2.internal.18e350487ef60f56  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-36.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-36.ec2.internal,},FirstTimestamp:2026-10-30 13:02:28.921511766 +0000 UTC m=+0.355510701,LastTimestamp:2026-10-30 13:02:28.921511766 +0000 UTC m=+0.355510701,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-36.ec2.internal,}"
Oct 30 13:02:28.954510 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.954481    2574 policy_none.go:50] "Start"
Oct 30 13:02:28.954562 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.954513    2574 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 13:02:28.954562 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.954526    2574 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 13:02:28.956453 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.956442    2574 policy_none.go:44] "Start"
Oct 30 13:02:28.956712 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.956699    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 13:02:28.956755 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.956738    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tv9qc"
Oct 30 13:02:28.969157 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.969134    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Oct 30 13:02:28.969251 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.969125    2574 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-36.ec2.internal.18e3504880ba2f01  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-36.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-136-36.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-136-36.ec2.internal,},FirstTimestamp:2026-10-30 13:02:28.951142145 +0000 UTC m=+0.385141091,LastTimestamp:2026-10-30 13:02:28.951142145 +0000 UTC m=+0.385141091,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-36.ec2.internal,}"
Oct 30 13:02:28.969323 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.969286    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 13:02:28.982180 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.982154    2574 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.990740    2574 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.994030    2574 manager.go:341] "Starting Device Plugin manager"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.994791    2574 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.994802    2574 server.go:87] "Starting device plugin registration server"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.995073    2574 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.995084    2574 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.995196    2574 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.995282    2574 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:28.995290    2574 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.995784    2574 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:28.995828    2574 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.004976    2574 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.004996    2574 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.005012    2574 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 13:02:29.015393 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.005043    2574 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Oct 30 13:02:29.017590 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.017189    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:29.095638 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.095516    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:29.096477 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.096458    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:29.096615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.096507    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:29.096615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.096524    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:29.096615 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.096556    2574 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.105835 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.105637    2574 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"]
Oct 30 13:02:29.105902 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.105890    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:29.106738 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.106721    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:29.106811 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.106752    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:29.106811 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.106766    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:29.108862 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.108848    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:29.109003 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.108986    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.109047 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.109018    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:29.109624 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.109609    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:29.109702 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.109638    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:29.109702 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.109652    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:29.109702 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.109611    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:29.109807 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.109708    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:29.109807 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.109720    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:29.111485 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.111472    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.111567 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.111514    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:29.112159 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.112146    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:29.112230 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.112170    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:29.112230 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.112186    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:29.113510 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.113480    2574 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.113585 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.113515    2574 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-36.ec2.internal\": node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.132889 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.132864    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3da04c9aa773780d1e2c9af45a694726-config\") pod \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" (UID: \"3da04c9aa773780d1e2c9af45a694726\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.144855 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.144838    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-36.ec2.internal\" not found" node="ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.147978 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.147964    2574 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-36.ec2.internal\" not found" node="ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.153427 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.153408    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.233066 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.233040    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3da04c9aa773780d1e2c9af45a694726-config\") pod \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" (UID: \"3da04c9aa773780d1e2c9af45a694726\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.233225 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.233071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.233225 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.233101    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.233225 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.233160    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3da04c9aa773780d1e2c9af45a694726-config\") pod \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" (UID: \"3da04c9aa773780d1e2c9af45a694726\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.254393 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.254368    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.333323 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.333292    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.333323 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.333324    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.333486 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.333371    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.333486 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.333386    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.354909 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.354837    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.448410 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.448378    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.451329 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:29.451309    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:29.455533 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.455513    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.556598 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.556557    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.657191 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.657105    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:29.757575 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:29.757540    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:31.046244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:31.046218    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:31.050381 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.050359    2574 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 13:02:31.050519 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.050501    2574 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 13:02:31.147037 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:31.147012    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:31.149193 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.149144    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:57:30 +0000 UTC" deadline="2028-04-27 10:09:40.448270317 +0000 UTC"
Oct 30 13:02:31.149193 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.149190    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13077h7m9.299083617s"
Oct 30 13:02:31.211173 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.211142    2574 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:02:31.212047 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.212032    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:02:31.212120 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.212061    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:02:31.212120 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.212070    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:02:31.240873 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:31.240842    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da04c9aa773780d1e2c9af45a694726.slice/crio-9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35 WatchSource:0}: Error finding container 9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35: Status 404 returned error can't find the container with id 9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35
Oct 30 13:02:31.246535 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.246522    2574 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:02:31.247142 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:31.247124    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:31.269330 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:31.269308    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7120be761a1303883792456d28b86647.slice/crio-683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5 WatchSource:0}: Error finding container 683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5: Status 404 returned error can't find the container with id 683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5
Oct 30 13:02:31.348061 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:31.348029    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:31.369851 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.369835    2574 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:31.448333 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:31.448285    2574 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:02:31.467913 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.467897    2574 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:31.519955 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.519936    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:31.543637 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.543621    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:02:31.545392 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.545379    2574 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:02:31.567624 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.567605    2574 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:02:31.668201 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:31.668164    2574 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:32.095080 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.095053    2574 apiserver.go:52] "Watching apiserver"
Oct 30 13:02:32.120027 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.119996    2574 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 13:02:32.141093 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.141048    2574 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 13:02:32.141425 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.141403    2574 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal","openshift-image-registry/node-ca-gr5nl","openshift-multus/multus-additional-cni-plugins-c2nts","openshift-cluster-node-tuning-operator/tuned-k52kk","openshift-multus/network-metrics-daemon-66rk6","openshift-network-diagnostics/network-check-target-8542k","openshift-ovn-kubernetes/ovnkube-node-28lqx","kube-system/konnectivity-agent-xnlf9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg","openshift-dns/node-resolver-5sstj","openshift-multus/multus-lz7gh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal","openshift-network-operator/iptables-alerter-nlmnl","kube-system/global-pull-secret-syncer-j4n8s"]
Oct 30 13:02:32.144249 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.144222    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.147318 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.146653    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.149298 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.148932    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.150923 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.150902    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.154216 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.153173    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:32.154216 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.153246    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:32.155516 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.155497    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:32.155599 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.155558    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:32.158728 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.158159    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.160129 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.160106    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.160225 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.160155    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:32.161479 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.161460    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 13:02:32.161585 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.161572    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 13:02:32.162701 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.162369    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxrfm\""
Oct 30 13:02:32.162701 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.162475    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 13:02:32.162701 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.162567    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 13:02:32.162887 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.162759    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.164828 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.164809    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 13:02:32.165113 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.165091    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:32.167533 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.167513    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 13:02:32.169164 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.169148    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.171869 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.171845    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.173987 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.173969    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.175702 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.175324    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 13:02:32.175702 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.175494    2574 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 13:02:32.175702 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.175563    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m8jrt\""
Oct 30 13:02:32.175897 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.175736    2574 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-65mcm"
Oct 30 13:02:32.176031 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.176002    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:32.198669 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.198612    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal" event={"ID":"3da04c9aa773780d1e2c9af45a694726","Type":"ContainerStarted","Data":"9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35"}
Oct 30 13:02:32.200906 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.200884    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal" event={"ID":"7120be761a1303883792456d28b86647","Type":"ContainerStarted","Data":"683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5"}
Oct 30 13:02:32.221706 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.221690    2574 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 13:02:32.231486 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231460    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-konnectivity-ca\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.231595 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231502    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.231595 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231524    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.231595 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231547    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cnibin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.231595 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231569    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cni-binary-copy\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.231807 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231681    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnjj8\" (UniqueName: \"kubernetes.io/projected/9bc491f0-f83f-48fa-8f83-92f88f9f9467-kube-api-access-jnjj8\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.231807 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231708    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-systemd-units\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.231807 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231781    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.231924 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231816    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-config\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.231924 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231872    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovn-node-metrics-cert\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.231924 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231900    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysconfig\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.232039 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231934    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-systemd\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.232039 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231954    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-lib-modules\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.232039 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.231994    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8ww8\" (UniqueName: \"kubernetes.io/projected/ebb46e24-3980-4909-83fb-bf22d54b5031-kube-api-access-t8ww8\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:32.232039 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232033    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-sys-fs\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.232187 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232075    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-socket-dir-parent\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232187 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232109    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-etc-kubernetes\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232187 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232154    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-log-socket\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.232187 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232174    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-netd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.232335 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232215    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cc5462-99f5-410c-97d6-b414507f297a-tmp-dir\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.232335 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec2afa3-085d-4e46-8488-60154d80ec91-host\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.232335 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232265    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-conf-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232335 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232327    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzhw\" (UniqueName: \"kubernetes.io/projected/02f4d975-e0f4-456c-a85e-2a1389468f5c-kube-api-access-qjzhw\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.232475 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232358    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-bin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232475 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232394    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6l8z4\" (UniqueName: \"kubernetes.io/projected/723b2da3-c191-420c-b6d1-7f8a4a9806cd-kube-api-access-6l8z4\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.232475 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232452    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-modprobe-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.232601 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232479    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232601 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232520    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-kubelet\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232601 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232560    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-ovn\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.232767 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232611    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:32.232767 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232659    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-cnibin\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.232767 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232695    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.232767 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232722    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.232767 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232747    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-hostroot\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232817    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-multus-certs\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232842    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-slash\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232866    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232893    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-device-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232917    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x75s\" (UniqueName: \"kubernetes.io/projected/20cc5462-99f5-410c-97d6-b414507f297a-kube-api-access-6x75s\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232940    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4gz6\" (UniqueName: \"kubernetes.io/projected/b331a27e-e822-4419-b000-bf767335246a-kube-api-access-k4gz6\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232972    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-kubelet-config\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.232998 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.232996    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233039    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-multus\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233065    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-sys\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233088    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5zlw\" (UniqueName: \"kubernetes.io/projected/c56d84ee-39fa-48a9-9426-60b730cfaccc-kube-api-access-k5zlw\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233113    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-systemd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233140    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-host\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233163    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-system-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233185    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-netns\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233209    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-node-log\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233241    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-etc-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233264    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-bin\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233309    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-kubernetes\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233332    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-run\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233354    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-var-lib-kubelet\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233377    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-socket-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.233434 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233400    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-system-cni-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233492    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-k8s-cni-cncf-io\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233529    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-env-overrides\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233552    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233579    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233604    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b331a27e-e822-4419-b000-bf767335246a-iptables-alerter-script\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233628    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b331a27e-e822-4419-b000-bf767335246a-host-slash\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233669    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-dbus\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233693    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-netns\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233716    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-kubelet\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233748    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-var-lib-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233772    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-script-lib\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233795    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-conf\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233848    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-registration-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233874    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20cc5462-99f5-410c-97d6-b414507f297a-hosts-file\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233896    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ec2afa3-085d-4e46-8488-60154d80ec91-serviceca\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.234188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233928    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-agent-certs\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.234898 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233951    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-tuned\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.234898 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.233975    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.234898 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.234000    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-etc-selinux\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.234898 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.234022    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-tmp\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.234898 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.234047    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ftjq\" (UniqueName: \"kubernetes.io/projected/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kube-api-access-6ftjq\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.234898 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.234071    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9pbg\" (UniqueName: \"kubernetes.io/projected/7ec2afa3-085d-4e46-8488-60154d80ec91-kube-api-access-h9pbg\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.234898 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.234104    2574 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-daemon-config\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.263122 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.263099    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 13:02:32.263254 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.263236    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 13:02:32.263365 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.263330    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 13:02:32.263469 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.263447    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxh59\""
Oct 30 13:02:32.263919 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.263899    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8w9b7\""
Oct 30 13:02:32.263990 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.263952    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 13:02:32.263990 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.263968    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 13:02:32.264059 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264006    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 13:02:32.264091 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264066    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 13:02:32.264124 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264107    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 13:02:32.264456 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264444    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vz5h4\""
Oct 30 13:02:32.264503 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264494    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 13:02:32.264539 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264512    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 13:02:32.264539 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264531    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 13:02:32.264590 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264554    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 13:02:32.264590 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264567    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 13:02:32.264668 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264591    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:02:32.264668 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264604    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 13:02:32.264668 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264650    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-prxcs\""
Oct 30 13:02:32.264668 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264665    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrlh9\""
Oct 30 13:02:32.264786 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264696    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 13:02:32.264786 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264705    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 13:02:32.264786 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264731    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r278v\""
Oct 30 13:02:32.264786 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264752    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 13:02:32.264786 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.264786    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbfrs\""
Oct 30 13:02:32.306189 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.306166    2574 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:02:32.334792 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.334549    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8z4\" (UniqueName: \"kubernetes.io/projected/723b2da3-c191-420c-b6d1-7f8a4a9806cd-kube-api-access-6l8z4\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.334792 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.334591    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-modprobe-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.334792 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.334617    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.334792 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.334651    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-kubelet\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.334792 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.334675    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-ovn\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.334792 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.334697    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:32.335138 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335103    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-cnibin\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.335188 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.335237 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335192    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.335237 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335216    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.335333 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335249    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-hostroot\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.335333 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335275    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-multus-certs\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.335333 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335306    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-slash\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.335333 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335331    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.335501 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335351    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-device-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.335501 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335370    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x75s\" (UniqueName: \"kubernetes.io/projected/20cc5462-99f5-410c-97d6-b414507f297a-kube-api-access-6x75s\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.335501 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gz6\" (UniqueName: \"kubernetes.io/projected/b331a27e-e822-4419-b000-bf767335246a-kube-api-access-k4gz6\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.335501 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335425    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-kubelet-config\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.335501 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335455    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.335501 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335490    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-multus\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.335783 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335523    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-sys\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.335783 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335669    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-modprobe-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.335877 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335845    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-ovn\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.335940 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335923    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.335990 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.335954    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:32.335990 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335969    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-kubelet\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.336071 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.336058    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs podName:ebb46e24-3980-4909-83fb-bf22d54b5031 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:32.836028441 +0000 UTC m=+3.081882382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs") pod "network-metrics-daemon-66rk6" (UID: "ebb46e24-3980-4909-83fb-bf22d54b5031") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:32.336129 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.336116    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-cnibin\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.336397 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.336376    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.336997 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.336976    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.337089 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337075    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-hostroot\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.337140 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337105    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-kubelet-config\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.337253 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.337239    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:32.337318 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.337297    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret podName:37b49c3d-d533-4d1f-898b-2c2fa83e2126 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:32.837274356 +0000 UTC m=+3.083128296 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret") pod "global-pull-secret-syncer-j4n8s" (UID: "37b49c3d-d533-4d1f-898b-2c2fa83e2126") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:32.337405 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337376    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-multus-certs\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.337471 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337457    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-sys\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.337522 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.335546    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zlw\" (UniqueName: \"kubernetes.io/projected/c56d84ee-39fa-48a9-9426-60b730cfaccc-kube-api-access-k5zlw\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.337572 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337550    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-slash\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.337572 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337562    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-systemd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.337671 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337596    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-host\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.337671 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337614    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-multus\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.337671 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337621    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-system-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.337805 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337670    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-netns\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.337805 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337676    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.337805 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337721    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-node-log\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.337805 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337735    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-host\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.337805 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337753    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-etc-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.337805 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337783    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-netns\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337841    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-bin\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337851    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-systemd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337794    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-bin\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337896    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-kubernetes\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337901    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-etc-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337928    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-run\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337957    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-node-log\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.337983    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-var-lib-kubelet\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338012    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-system-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-socket-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.338056 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338047    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-system-cni-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338067    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-run\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338073    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-k8s-cni-cncf-io\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338117    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-env-overrides\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338115    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-k8s-cni-cncf-io\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338173    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338254    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338286    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b331a27e-e822-4419-b000-bf767335246a-iptables-alerter-script\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338312    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b331a27e-e822-4419-b000-bf767335246a-host-slash\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338354    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-dbus\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338392    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-netns\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338448    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-kubelet\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.338509 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338478    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-var-lib-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338522    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-script-lib\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338572    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-conf\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338601    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-registration-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338624    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20cc5462-99f5-410c-97d6-b414507f297a-hosts-file\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338663    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ec2afa3-085d-4e46-8488-60154d80ec91-serviceca\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338693    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-agent-certs\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338719    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-tuned\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338749    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338784    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-etc-selinux\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338806    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-tmp\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338825    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ftjq\" (UniqueName: \"kubernetes.io/projected/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kube-api-access-6ftjq\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338843    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pbg\" (UniqueName: \"kubernetes.io/projected/7ec2afa3-085d-4e46-8488-60154d80ec91-kube-api-access-h9pbg\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338878    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-daemon-config\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338911    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-konnectivity-ca\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338925    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.338950    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.339028 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339026    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cnibin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339057    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cni-binary-copy\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339091    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjj8\" (UniqueName: \"kubernetes.io/projected/9bc491f0-f83f-48fa-8f83-92f88f9f9467-kube-api-access-jnjj8\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339128    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-systemd-units\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339166    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339206    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-config\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339235    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovn-node-metrics-cert\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339261    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-env-overrides\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339258    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysconfig\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339297    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-systemd\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339325    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-lib-modules\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339376    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ww8\" (UniqueName: \"kubernetes.io/projected/ebb46e24-3980-4909-83fb-bf22d54b5031-kube-api-access-t8ww8\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339415    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-sys-fs\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339462    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-socket-dir-parent\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339502    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-etc-kubernetes\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339532    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-log-socket\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339574    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-netd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339574    2574 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 13:02:32.339780 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339687    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cc5462-99f5-410c-97d6-b414507f297a-tmp-dir\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339727    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec2afa3-085d-4e46-8488-60154d80ec91-host\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339763    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-conf-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339793    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzhw\" (UniqueName: \"kubernetes.io/projected/02f4d975-e0f4-456c-a85e-2a1389468f5c-kube-api-access-qjzhw\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339800    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-var-lib-kubelet\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339845    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-bin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339930    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-bin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.339965    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-system-cni-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340066    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340229    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340304    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-dbus\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340336    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b331a27e-e822-4419-b000-bf767335246a-host-slash\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340363    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-netns\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340405    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-kubelet\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.340523 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340430    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-var-lib-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.341193 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340680    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-script-lib\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.341193 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340818    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-conf\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.341193 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340856    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-device-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.341193 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.340955    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-kubernetes\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.341385 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341191    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-etc-selinux\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.341385 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341275    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-registration-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.341385 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341322    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysconfig\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.341385 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341360    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ec2afa3-085d-4e46-8488-60154d80ec91-serviceca\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.341569 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341462    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20cc5462-99f5-410c-97d6-b414507f297a-hosts-file\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.341569 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341537    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-log-socket\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.341684 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341609    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-systemd\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.341783 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.341717    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cnibin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.342126 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342087    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-netd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.342226 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342208    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cni-binary-copy\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.342297 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342211    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-config\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.342367 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342308    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-sys-fs\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.342438 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342328    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.342438 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342389    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-socket-dir-parent\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.342550 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342459    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-conf-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.342550 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342461    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-systemd-units\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342827    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cc5462-99f5-410c-97d6-b414507f297a-tmp-dir\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342861    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-socket-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342947    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec2afa3-085d-4e46-8488-60154d80ec91-host\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.342996    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.343080    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-lib-modules\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.343096    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.343175    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-konnectivity-ca\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.343314    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-etc-kubernetes\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.343399    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b331a27e-e822-4419-b000-bf767335246a-iptables-alerter-script\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.343581 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.343490    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-daemon-config\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.344740 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.344698    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-tuned\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.345604 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.345498    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovn-node-metrics-cert\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.346444 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.346418    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-agent-certs\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.348950 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.348910    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-tmp\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.353488 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.353465    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.356831 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.356779    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8z4\" (UniqueName: \"kubernetes.io/projected/723b2da3-c191-420c-b6d1-7f8a4a9806cd-kube-api-access-6l8z4\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.357313 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.357284    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zlw\" (UniqueName: \"kubernetes.io/projected/c56d84ee-39fa-48a9-9426-60b730cfaccc-kube-api-access-k5zlw\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.357515 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.357494    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gz6\" (UniqueName: \"kubernetes.io/projected/b331a27e-e822-4419-b000-bf767335246a-kube-api-access-k4gz6\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.360974 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.360954    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:32.360974 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.360977    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:32.361127 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.360988    2574 projected.go:196] Error preparing data for projected volume kube-api-access-gnz2d for pod openshift-network-diagnostics/network-check-target-8542k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:32.361127 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.361035    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d podName:28913fb0-2984-4375-8fe0-e15682bde559 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:32.861019186 +0000 UTC m=+3.106873110 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz2d" (UniqueName: "kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d") pod "network-check-target-8542k" (UID: "28913fb0-2984-4375-8fe0-e15682bde559") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:32.362106 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.362054    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x75s\" (UniqueName: \"kubernetes.io/projected/20cc5462-99f5-410c-97d6-b414507f297a-kube-api-access-6x75s\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.363485 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.363118    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzhw\" (UniqueName: \"kubernetes.io/projected/02f4d975-e0f4-456c-a85e-2a1389468f5c-kube-api-access-qjzhw\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.365822 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.365796    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjj8\" (UniqueName: \"kubernetes.io/projected/9bc491f0-f83f-48fa-8f83-92f88f9f9467-kube-api-access-jnjj8\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.366842 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.366803    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pbg\" (UniqueName: \"kubernetes.io/projected/7ec2afa3-085d-4e46-8488-60154d80ec91-kube-api-access-h9pbg\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.367252 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.367234    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ww8\" (UniqueName: \"kubernetes.io/projected/ebb46e24-3980-4909-83fb-bf22d54b5031-kube-api-access-t8ww8\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:32.367394 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.367374    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ftjq\" (UniqueName: \"kubernetes.io/projected/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kube-api-access-6ftjq\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.478574 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.478546    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:02:32.484946 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.484925    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:32.488447 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.488428    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:02:32.497791 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.497771    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:02:32.505712 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.505694    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:02:32.507847 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.507825    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:02:32.508826 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:32.508800    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb331a27e_e822_4419_b000_bf767335246a.slice/crio-c60a10900bae5fc4c2ec5d7a6530b5ce0b239d539e0c7c24f3e09088f6ea01a4 WatchSource:0}: Error finding container c60a10900bae5fc4c2ec5d7a6530b5ce0b239d539e0c7c24f3e09088f6ea01a4: Status 404 returned error can't find the container with id c60a10900bae5fc4c2ec5d7a6530b5ce0b239d539e0c7c24f3e09088f6ea01a4
Oct 30 13:02:32.509609 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.509567    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:02:32.517583 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.517561    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lz7gh"
Oct 30 13:02:32.517885 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:32.517862    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56d84ee_39fa_48a9_9426_60b730cfaccc.slice/crio-0ef2714f07b7db26e62dadf1b0965ccda8eaa622396cb1bdb0dd91719518bf9a WatchSource:0}: Error finding container 0ef2714f07b7db26e62dadf1b0965ccda8eaa622396cb1bdb0dd91719518bf9a: Status 404 returned error can't find the container with id 0ef2714f07b7db26e62dadf1b0965ccda8eaa622396cb1bdb0dd91719518bf9a
Oct 30 13:02:32.518665 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:32.518623    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod723b2da3_c191_420c_b6d1_7f8a4a9806cd.slice/crio-a73423083f2ae4dfca58d389e5d518e9b727b6f4039621c31c186ef8fd09d367 WatchSource:0}: Error finding container a73423083f2ae4dfca58d389e5d518e9b727b6f4039621c31c186ef8fd09d367: Status 404 returned error can't find the container with id a73423083f2ae4dfca58d389e5d518e9b727b6f4039621c31c186ef8fd09d367
Oct 30 13:02:32.524105 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.524079    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:32.526453 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:32.526430    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f4d975_e0f4_456c_a85e_2a1389468f5c.slice/crio-3e968d877a5022c9878e52155f0e8cd982671071ab9d3c304c8b006fd4d16cb7 WatchSource:0}: Error finding container 3e968d877a5022c9878e52155f0e8cd982671071ab9d3c304c8b006fd4d16cb7: Status 404 returned error can't find the container with id 3e968d877a5022c9878e52155f0e8cd982671071ab9d3c304c8b006fd4d16cb7
Oct 30 13:02:32.527878 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:32.527086    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda470b83_8d33_4fe6_a4c9_2c3a0b85d060.slice/crio-c0789d6ded7a57a3e444a83cdc0b9b998ebb97ca6cc86ddd447deb4ac8fa13c9 WatchSource:0}: Error finding container c0789d6ded7a57a3e444a83cdc0b9b998ebb97ca6cc86ddd447deb4ac8fa13c9: Status 404 returned error can't find the container with id c0789d6ded7a57a3e444a83cdc0b9b998ebb97ca6cc86ddd447deb4ac8fa13c9
Oct 30 13:02:32.536816 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:32.536795    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bc491f0_f83f_48fa_8f83_92f88f9f9467.slice/crio-d2e1a863f07002790a0c7b8f2479780309a75ae368f87cc80e15c33608b77a50 WatchSource:0}: Error finding container d2e1a863f07002790a0c7b8f2479780309a75ae368f87cc80e15c33608b77a50: Status 404 returned error can't find the container with id d2e1a863f07002790a0c7b8f2479780309a75ae368f87cc80e15c33608b77a50
Oct 30 13:02:32.539990 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:02:32.539948    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a2b2f25_b784_40c6_8fcc_aa4fca8ed8e3.slice/crio-262a20c7f962dbf5f51a88684f98cad8effd2231a8fabbc0eccb68edbc3e2ce6 WatchSource:0}: Error finding container 262a20c7f962dbf5f51a88684f98cad8effd2231a8fabbc0eccb68edbc3e2ce6: Status 404 returned error can't find the container with id 262a20c7f962dbf5f51a88684f98cad8effd2231a8fabbc0eccb68edbc3e2ce6
Oct 30 13:02:32.842790 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.842592    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:32.842939 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.842817    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:32.842939 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.842761    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:32.842939 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.842921    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs podName:ebb46e24-3980-4909-83fb-bf22d54b5031 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:33.842899575 +0000 UTC m=+4.088753516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs") pod "network-metrics-daemon-66rk6" (UID: "ebb46e24-3980-4909-83fb-bf22d54b5031") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:32.843104 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.842943    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:32.843104 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.842985    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret podName:37b49c3d-d533-4d1f-898b-2c2fa83e2126 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:33.842972142 +0000 UTC m=+4.088826069 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret") pod "global-pull-secret-syncer-j4n8s" (UID: "37b49c3d-d533-4d1f-898b-2c2fa83e2126") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:32.944785 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:32.944109    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:32.944785 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.944267    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:32.944785 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.944286    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:32.944785 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.944299    2574 projected.go:196] Error preparing data for projected volume kube-api-access-gnz2d for pod openshift-network-diagnostics/network-check-target-8542k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:32.944785 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:32.944353    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d podName:28913fb0-2984-4375-8fe0-e15682bde559 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:33.944337169 +0000 UTC m=+4.190191099 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz2d" (UniqueName: "kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d") pod "network-check-target-8542k" (UID: "28913fb0-2984-4375-8fe0-e15682bde559") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:33.179671 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.179544    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:32 +0000 UTC" deadline="2028-07-28 23:38:37.269351406 +0000 UTC"
Oct 30 13:02:33.179671 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.179599    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15298h36m4.089756755s"
Oct 30 13:02:33.211892 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.211800    2574 generic.go:356] "Generic (PLEG): container finished" podID="7120be761a1303883792456d28b86647" containerID="48c341ed88af414ef642db376cdb62c95960ebb5efdb043b814a6b0a779ca789" exitCode=0
Oct 30 13:02:33.212047 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.211894    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal" event={"ID":"7120be761a1303883792456d28b86647","Type":"ContainerDied","Data":"48c341ed88af414ef642db376cdb62c95960ebb5efdb043b814a6b0a779ca789"}
Oct 30 13:02:33.228853 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.228823    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnlf9" event={"ID":"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3","Type":"ContainerStarted","Data":"262a20c7f962dbf5f51a88684f98cad8effd2231a8fabbc0eccb68edbc3e2ce6"}
Oct 30 13:02:33.251069 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.251010    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lz7gh" event={"ID":"9bc491f0-f83f-48fa-8f83-92f88f9f9467","Type":"ContainerStarted","Data":"d2e1a863f07002790a0c7b8f2479780309a75ae368f87cc80e15c33608b77a50"}
Oct 30 13:02:33.274287 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.274257    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5sstj" event={"ID":"20cc5462-99f5-410c-97d6-b414507f297a","Type":"ContainerStarted","Data":"b1ed60384455e228b7939b788f2e262172466aecbb83bc89f36ed6b455f0a2d7"}
Oct 30 13:02:33.277365 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.277328    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gr5nl" event={"ID":"7ec2afa3-085d-4e46-8488-60154d80ec91","Type":"ContainerStarted","Data":"3e83be51b4dbf0c424325b0a7fd4841789fe2010d268b780e7b692dfd15a27bc"}
Oct 30 13:02:33.279558 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.279533    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerStarted","Data":"3e968d877a5022c9878e52155f0e8cd982671071ab9d3c304c8b006fd4d16cb7"}
Oct 30 13:02:33.281887 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.281856    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"a73423083f2ae4dfca58d389e5d518e9b727b6f4039621c31c186ef8fd09d367"}
Oct 30 13:02:33.294144 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.294114    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal" event={"ID":"3da04c9aa773780d1e2c9af45a694726","Type":"ContainerStarted","Data":"ab2ae0fc353b172945c7fc9135be4b100df0f4e504995491127d34c315b7ce62"}
Oct 30 13:02:33.298703 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.298676    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg" event={"ID":"da470b83-8d33-4fe6-a4c9-2c3a0b85d060","Type":"ContainerStarted","Data":"c0789d6ded7a57a3e444a83cdc0b9b998ebb97ca6cc86ddd447deb4ac8fa13c9"}
Oct 30 13:02:33.303777 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.303753    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k52kk" event={"ID":"c56d84ee-39fa-48a9-9426-60b730cfaccc","Type":"ContainerStarted","Data":"0ef2714f07b7db26e62dadf1b0965ccda8eaa622396cb1bdb0dd91719518bf9a"}
Oct 30 13:02:33.344821 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.344786    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nlmnl" event={"ID":"b331a27e-e822-4419-b000-bf767335246a","Type":"ContainerStarted","Data":"c60a10900bae5fc4c2ec5d7a6530b5ce0b239d539e0c7c24f3e09088f6ea01a4"}
Oct 30 13:02:33.347471 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.347416    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal" podStartSLOduration=2.347399645 podStartE2EDuration="2.347399645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:33.335206795 +0000 UTC m=+3.581060741" watchObservedRunningTime="2026-10-30 13:02:33.347399645 +0000 UTC m=+3.593253591"
Oct 30 13:02:33.851223 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.851142    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:33.851223 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.851193    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:33.851428 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.851309    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:33.851428 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.851363    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret podName:37b49c3d-d533-4d1f-898b-2c2fa83e2126 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.851346935 +0000 UTC m=+6.097200859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret") pod "global-pull-secret-syncer-j4n8s" (UID: "37b49c3d-d533-4d1f-898b-2c2fa83e2126") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:33.851428 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.851427    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:33.851582 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.851463    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs podName:ebb46e24-3980-4909-83fb-bf22d54b5031 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.851449273 +0000 UTC m=+6.097303207 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs") pod "network-metrics-daemon-66rk6" (UID: "ebb46e24-3980-4909-83fb-bf22d54b5031") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:33.952419 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:33.952384    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:33.952599 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.952553    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:33.952599 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.952574    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:33.952599 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.952585    2574 projected.go:196] Error preparing data for projected volume kube-api-access-gnz2d for pod openshift-network-diagnostics/network-check-target-8542k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:33.952781 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:33.952649    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d podName:28913fb0-2984-4375-8fe0-e15682bde559 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:35.952619481 +0000 UTC m=+6.198473419 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz2d" (UniqueName: "kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d") pod "network-check-target-8542k" (UID: "28913fb0-2984-4375-8fe0-e15682bde559") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:34.180679 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:34.180583    2574 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:32 +0000 UTC" deadline="2028-05-18 20:04:39.47980621 +0000 UTC"
Oct 30 13:02:34.180679 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:34.180623    2574 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13591h2m5.299187976s"
Oct 30 13:02:34.195620 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:34.195405    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:34.195620 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:34.195525    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:34.195874 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:34.195717    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:34.195874 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:34.195796    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:34.195874 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:34.195869    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:34.196012 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:34.195947    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:34.355120 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:34.355080    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal" event={"ID":"7120be761a1303883792456d28b86647","Type":"ContainerStarted","Data":"72506b471189f74f189e13c340d9dad43e26f29e8df7ee462350e67ad7446510"}
Oct 30 13:02:35.361483 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:35.361384    2574 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="b5706ee4fb747b6b343856e88e339569ed1a417c80b7a90090e3a65a6af80cf3" exitCode=0
Oct 30 13:02:35.362412 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:35.362386    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerDied","Data":"b5706ee4fb747b6b343856e88e339569ed1a417c80b7a90090e3a65a6af80cf3"}
Oct 30 13:02:35.436191 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:35.436123    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal" podStartSLOduration=4.436104747 podStartE2EDuration="4.436104747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:34.39659522 +0000 UTC m=+4.642449166" watchObservedRunningTime="2026-10-30 13:02:35.436104747 +0000 UTC m=+5.681958694"
Oct 30 13:02:35.868916 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:35.868161    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:35.868916 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:35.868220    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:35.868916 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.868375    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:35.868916 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.868434    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret podName:37b49c3d-d533-4d1f-898b-2c2fa83e2126 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:39.868416007 +0000 UTC m=+10.114269937 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret") pod "global-pull-secret-syncer-j4n8s" (UID: "37b49c3d-d533-4d1f-898b-2c2fa83e2126") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:35.868916 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.868837    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:35.868916 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.868884    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs podName:ebb46e24-3980-4909-83fb-bf22d54b5031 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:39.868868108 +0000 UTC m=+10.114722037 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs") pod "network-metrics-daemon-66rk6" (UID: "ebb46e24-3980-4909-83fb-bf22d54b5031") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:35.969333 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:35.968855    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:35.969333 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.968976    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:35.969333 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.968989    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:35.969333 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.968997    2574 projected.go:196] Error preparing data for projected volume kube-api-access-gnz2d for pod openshift-network-diagnostics/network-check-target-8542k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:35.969333 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:35.969034    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d podName:28913fb0-2984-4375-8fe0-e15682bde559 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:39.969024012 +0000 UTC m=+10.214877935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz2d" (UniqueName: "kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d") pod "network-check-target-8542k" (UID: "28913fb0-2984-4375-8fe0-e15682bde559") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:36.197190 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:36.196298    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:36.197190 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:36.196440    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:36.197190 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:36.196838    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:36.197190 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:36.196924    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:36.197190 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:36.196982    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:36.197190 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:36.197055    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:37.372175 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:37.372142    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xnlf9" event={"ID":"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3","Type":"ContainerStarted","Data":"6da5377aaef4d93fd0d54a9494011c9c85664b747220e5bd118f38a77f11f6aa"}
Oct 30 13:02:38.195814 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:38.195788    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:38.195931 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:38.195910    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:38.195990 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:38.195981    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:38.196078 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:38.196061    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:38.196135 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:38.196116    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:38.196199 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:38.196180    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:38.377275 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:38.377232    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg" event={"ID":"da470b83-8d33-4fe6-a4c9-2c3a0b85d060","Type":"ContainerStarted","Data":"2d1c68d323d3956bd502a3d7aab4fb5b1849e3cce4827f24d1ef2bb4cf4b760c"}
Oct 30 13:02:38.380234 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:38.380204    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gr5nl" event={"ID":"7ec2afa3-085d-4e46-8488-60154d80ec91","Type":"ContainerStarted","Data":"225aad66ba0cdd0800c24e711d40beb9af4545037c6114303f234591c19a8fed"}
Oct 30 13:02:38.419752 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:38.419676    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xnlf9" podStartSLOduration=4.560060605 podStartE2EDuration="8.419659121s" totalImagesPullingTime="3.859598516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:37.413781667 +0000 UTC m=+7.659635611" watchObservedRunningTime="2026-10-30 13:02:38.419659121 +0000 UTC m=+8.665513066"
Oct 30 13:02:38.607153 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:38.607076    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:39.692939 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:39.692911    2574 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 13:02:39.899656 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:39.899613    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:39.899751 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:39.899681    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:39.899841 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:39.899828    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:39.899898 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:39.899887    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret podName:37b49c3d-d533-4d1f-898b-2c2fa83e2126 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:47.899869735 +0000 UTC m=+18.145723665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret") pod "global-pull-secret-syncer-j4n8s" (UID: "37b49c3d-d533-4d1f-898b-2c2fa83e2126") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:39.900191 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:39.900176    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:39.900231 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:39.900224    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs podName:ebb46e24-3980-4909-83fb-bf22d54b5031 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:47.900209558 +0000 UTC m=+18.146063505 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs") pod "network-metrics-daemon-66rk6" (UID: "ebb46e24-3980-4909-83fb-bf22d54b5031") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:40.001244 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.000924    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:40.001244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:40.001114    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:40.001244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:40.001132    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:40.001244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:40.001146    2574 projected.go:196] Error preparing data for projected volume kube-api-access-gnz2d for pod openshift-network-diagnostics/network-check-target-8542k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:40.001244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:40.001209    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d podName:28913fb0-2984-4375-8fe0-e15682bde559 nodeName:}" failed. No retries permitted until 2026-10-30 13:02:48.001190952 +0000 UTC m=+18.247044877 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz2d" (UniqueName: "kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d") pod "network-check-target-8542k" (UID: "28913fb0-2984-4375-8fe0-e15682bde559") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:40.196161 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.196137    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:40.196322 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:40.196299    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:40.197116 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.196741    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:40.197116 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.196758    2574 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:02:39.692935158Z","UUID":"56382b44-0c41-4b8f-98a8-556e02e4d702","Handler":null,"Name":"","Endpoint":""}
Oct 30 13:02:40.197116 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:40.196920    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:40.197116 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.196974    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:40.197116 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:40.197080    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:40.199684 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.199663    2574 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 13:02:40.199771 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.199705    2574 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 13:02:40.387415 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.387382    2574 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="f13ae91d0c2caaa3e879c1994a5ecb46a1c445b40be2e5edbfecbcdd60d0668b" exitCode=0
Oct 30 13:02:40.387541 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.387471    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerDied","Data":"f13ae91d0c2caaa3e879c1994a5ecb46a1c445b40be2e5edbfecbcdd60d0668b"}
Oct 30 13:02:40.393205 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.392928    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg" event={"ID":"da470b83-8d33-4fe6-a4c9-2c3a0b85d060","Type":"ContainerStarted","Data":"8ecc4efb1300ad00633d3afd19cf2a91a3a38a7601f86f8840113516eb3afe2a"}
Oct 30 13:02:40.395575 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.395089    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lz7gh" event={"ID":"9bc491f0-f83f-48fa-8f83-92f88f9f9467","Type":"ContainerStarted","Data":"c8223c2eaa2764ac57286564f5ed350df8ab750b34ebd3c37279330044cb578a"}
Oct 30 13:02:40.436798 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:40.436175    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gr5nl" podStartSLOduration=5.567134335 podStartE2EDuration="10.436157175s" totalImagesPullingTime="4.86902284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:38.420622558 +0000 UTC m=+8.666476505" watchObservedRunningTime="2026-10-30 13:02:40.436157175 +0000 UTC m=+10.682011123"
Oct 30 13:02:41.400321 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.400245    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg" event={"ID":"da470b83-8d33-4fe6-a4c9-2c3a0b85d060","Type":"ContainerStarted","Data":"9cf5a3cce9d17557e63162c23360d5d266274af2d1399af3f44b214f43148e61"}
Oct 30 13:02:41.402293 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.402260    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5sstj" event={"ID":"20cc5462-99f5-410c-97d6-b414507f297a","Type":"ContainerStarted","Data":"6d16dab43931e47bf948c863018700ddcaab8215cfd5c9641ee6e0dc80034e5e"}
Oct 30 13:02:41.405099 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.405071    2574 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="62678e4f2255cfcc5a3c37362143e49cc8b330dc970973216a9a1047ccb8a62a" exitCode=0
Oct 30 13:02:41.405815 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.405795    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerDied","Data":"62678e4f2255cfcc5a3c37362143e49cc8b330dc970973216a9a1047ccb8a62a"}
Oct 30 13:02:41.442859 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.442813    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-lz7gh" podStartSLOduration=4.138354445 podStartE2EDuration="11.442801326s" totalImagesPullingTime="7.304446881s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:40.477768587 +0000 UTC m=+10.723622535" watchObservedRunningTime="2026-10-30 13:02:41.442801326 +0000 UTC m=+11.688655290"
Oct 30 13:02:41.488531 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.488490    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg" podStartSLOduration=3.773593181 podStartE2EDuration="11.488477798s" totalImagesPullingTime="7.714884617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:41.443231488 +0000 UTC m=+11.689085437" watchObservedRunningTime="2026-10-30 13:02:41.488477798 +0000 UTC m=+11.734331745"
Oct 30 13:02:41.528601 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.528552    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-5sstj" podStartSLOduration=3.773544584 podStartE2EDuration="11.528540185s" totalImagesPullingTime="7.754995601s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:41.528112313 +0000 UTC m=+11.773966262" watchObservedRunningTime="2026-10-30 13:02:41.528540185 +0000 UTC m=+11.774394131"
Oct 30 13:02:41.705249 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.705179    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:41.705890 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:41.705709    2574 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:42.195081 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:42.194340    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:42.195081 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:42.194390    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:42.195081 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:42.194488    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:42.195081 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:42.194886    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:42.195081 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:42.194945    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:42.195081 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:42.195007    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:42.408947 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:42.408859    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nlmnl" event={"ID":"b331a27e-e822-4419-b000-bf767335246a","Type":"ContainerStarted","Data":"7791ab9f0730000cccefecdec636d09f1d5470ee04aa2d3748e838855399f4e5"}
Oct 30 13:02:42.409717 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:42.409697    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:02:42.449237 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:42.449123    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nlmnl" podStartSLOduration=4.783322544 podStartE2EDuration="12.449108764s" totalImagesPullingTime="7.66578622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:42.449080044 +0000 UTC m=+12.694933991" watchObservedRunningTime="2026-10-30 13:02:42.449108764 +0000 UTC m=+12.694962709"
Oct 30 13:02:43.413546 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:43.413502    2574 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="3827684666aeabfe3bf39e3fb8c08b4210fac67df9f172aac5740c6890498d25" exitCode=0
Oct 30 13:02:43.414266 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:43.413584    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerDied","Data":"3827684666aeabfe3bf39e3fb8c08b4210fac67df9f172aac5740c6890498d25"}
Oct 30 13:02:43.415144 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:43.415116    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k52kk" event={"ID":"c56d84ee-39fa-48a9-9426-60b730cfaccc","Type":"ContainerStarted","Data":"4f24d4a340135f76cd52eaa5337aba82d6bc94aa543fc94a09d614332339e130"}
Oct 30 13:02:43.530194 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:43.530142    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k52kk" podStartSLOduration=3.4412648949999998 podStartE2EDuration="13.530129033s" totalImagesPullingTime="10.088864138s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:43.529802215 +0000 UTC m=+13.775656162" watchObservedRunningTime="2026-10-30 13:02:43.530129033 +0000 UTC m=+13.775982978"
Oct 30 13:02:44.194004 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:44.193971    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:44.194181 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:44.193986    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:44.194181 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:44.194095    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:44.194181 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:44.194109    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:44.194351 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:44.194200    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:44.194351 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:44.194275    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:45.423199 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:45.423008    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerStarted","Data":"bc6927950a222704000c9289725521955ae6cd84ec3744a57edc10bce522fcb8"}
Oct 30 13:02:46.196198 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:46.196170    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:46.196333 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:46.196243    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:46.196333 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:46.196265    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:46.196391 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:46.196346    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:46.196391 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:46.196379    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:46.196489 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:46.196470    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:46.427551 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:46.427510    2574 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="bc6927950a222704000c9289725521955ae6cd84ec3744a57edc10bce522fcb8" exitCode=0
Oct 30 13:02:46.427921 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:46.427576    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerDied","Data":"bc6927950a222704000c9289725521955ae6cd84ec3744a57edc10bce522fcb8"}
Oct 30 13:02:47.432971 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.432724    2574 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="6618dd61e2ba7ada7d935c42029111d5bc0cceeac44ce7fd8ee786de1956c6ab" exitCode=0
Oct 30 13:02:47.433445 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.432799    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerDied","Data":"6618dd61e2ba7ada7d935c42029111d5bc0cceeac44ce7fd8ee786de1956c6ab"}
Oct 30 13:02:47.436062 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.436040    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"295b476ce462b5926dc81f038417a4ada7b9a22cb9ed95362966e6fb776beed1"}
Oct 30 13:02:47.436155 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.436118    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"5895413e12064461c1069b2430ecc05b871621061527191676be88f38982bd68"}
Oct 30 13:02:47.436210 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.436165    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"9471968b16bdb461b399e4c7d0d07c4bf5c6ac56f2adb1c605c2359d7d27ca2b"}
Oct 30 13:02:47.436260 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.436215    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"14738228fa4de4ac1fbcd0a13e42e67314c1f7e60a1209b11a1a47650e2fe91d"}
Oct 30 13:02:47.436309 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.436261    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"f363dbcc499c39fe8db3e4b9ffb64bec0a9ad03751be15fc497dbea52df4bbb8"}
Oct 30 13:02:47.436361 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.436310    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"b856fa887ea478e99cda1d2c18679959c6a45dc265878c604bb1d7a61a5eef0d"}
Oct 30 13:02:47.955455 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.955423    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:47.955591 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:47.955466    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:47.955591 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:47.955561    2574 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:47.955591 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:47.955562    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:47.955734 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:47.955611    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret podName:37b49c3d-d533-4d1f-898b-2c2fa83e2126 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.955598088 +0000 UTC m=+34.201452011 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret") pod "global-pull-secret-syncer-j4n8s" (UID: "37b49c3d-d533-4d1f-898b-2c2fa83e2126") : object "kube-system"/"original-pull-secret" not registered
Oct 30 13:02:47.955734 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:47.955624    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs podName:ebb46e24-3980-4909-83fb-bf22d54b5031 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:03.955617674 +0000 UTC m=+34.201471599 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs") pod "network-metrics-daemon-66rk6" (UID: "ebb46e24-3980-4909-83fb-bf22d54b5031") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 13:02:48.056051 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:48.056015    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:48.056193 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:48.056170    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 13:02:48.056193 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:48.056189    2574 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 13:02:48.056287 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:48.056199    2574 projected.go:196] Error preparing data for projected volume kube-api-access-gnz2d for pod openshift-network-diagnostics/network-check-target-8542k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:48.056287 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:48.056251    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d podName:28913fb0-2984-4375-8fe0-e15682bde559 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:04.056234079 +0000 UTC m=+34.302088020 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gnz2d" (UniqueName: "kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d") pod "network-check-target-8542k" (UID: "28913fb0-2984-4375-8fe0-e15682bde559") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 13:02:48.196756 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:48.196718    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:48.196883 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:48.196788    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:48.196883 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:48.196811    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:48.196939 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:48.196891    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:48.196939 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:48.196927    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:48.197036 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:48.196965    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:48.441662 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:48.441573    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-c2nts" event={"ID":"02f4d975-e0f4-456c-a85e-2a1389468f5c","Type":"ContainerStarted","Data":"1b95809eff33ca44376a7ee232d15391e8980d31b55ead73245b1ff94315aafd"}
Oct 30 13:02:48.489203 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:48.489149    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-c2nts" podStartSLOduration=8.150292391 podStartE2EDuration="18.489136041s" totalImagesPullingTime="10.33884365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:48.488500121 +0000 UTC m=+18.734354067" watchObservedRunningTime="2026-10-30 13:02:48.489136041 +0000 UTC m=+18.734989987"
Oct 30 13:02:49.447387 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:49.447309    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"ec75780fc79ab89b530f3873cbee266c8560a11c90c2854e3e61d80da69c04af"}
Oct 30 13:02:50.198404 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:50.198370    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:50.198542 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:50.198416    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:50.198542 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:50.198465    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:50.198542 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:50.198484    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:50.198667 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:50.198540    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:50.198667 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:50.198596    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:51.453989 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:51.453674    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" event={"ID":"723b2da3-c191-420c-b6d1-7f8a4a9806cd","Type":"ContainerStarted","Data":"814c391eb693d88a543bf307bbcf4b7bce34264369bc46b3c0bfb52fc9a6cb0f"}
Oct 30 13:02:51.453989 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:51.453981    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:51.454452 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:51.454035    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:51.469019 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:51.469000    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:51.547416 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:51.547373    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx" podStartSLOduration=7.409389088 podStartE2EDuration="21.547362267s" totalImagesPullingTime="14.137973179s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:02:51.546976926 +0000 UTC m=+21.792830872" watchObservedRunningTime="2026-10-30 13:02:51.547362267 +0000 UTC m=+21.793216215"
Oct 30 13:02:52.196148 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:52.196116    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:52.196440 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:52.196199    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:52.196440 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:52.196297    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:52.196440 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:52.196314    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:52.196440 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:52.196367    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:52.196440 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:52.196406    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:52.456488 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:52.456408    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:52.470920 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:52.470898    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:02:53.158565 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:53.158508    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8542k"]
Oct 30 13:02:53.158708 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:53.158599    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:53.158708 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:53.158698    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:53.162236 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:53.162212    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4n8s"]
Oct 30 13:02:53.162342 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:53.162293    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:53.162378 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:53.162366    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:53.162677 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:53.162659    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-66rk6"]
Oct 30 13:02:53.162746 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:53.162736    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:53.162832 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:53.162818    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:55.193965 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:55.193938    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:55.194359 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:55.193935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:55.194359 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:55.194048    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:55.194359 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:55.193935    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:55.194359 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:55.194114    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:55.194359 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:55.194169    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:57.193928 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:57.193904    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:57.194244 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:57.193948    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:57.194244 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:57.193959    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:57.194244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:57.194026    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:57.194244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:57.194112    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:57.194244 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:57.194177    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:58.577590 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:58.577553    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5sstj_20cc5462-99f5-410c-97d6-b414507f297a/dns-node-resolver/0.log"
Oct 30 13:02:59.193798 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:59.193606    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:02:59.193942 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:59.193607    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:02:59.193942 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:59.193877    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66rk6" podUID="ebb46e24-3980-4909-83fb-bf22d54b5031"
Oct 30 13:02:59.194075 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:59.193966    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j4n8s" podUID="37b49c3d-d533-4d1f-898b-2c2fa83e2126"
Oct 30 13:02:59.194075 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:59.193627    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:02:59.196686 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:02:59.194063    2574 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8542k" podUID="28913fb0-2984-4375-8fe0-e15682bde559"
Oct 30 13:02:59.523328 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:59.523260    2574 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeReady"
Oct 30 13:02:59.523454 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:59.523379    2574 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 13:02:59.921776 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:02:59.921705    2574 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gr5nl_7ec2afa3-085d-4e46-8488-60154d80ec91/node-ca/0.log"
Oct 30 13:03:01.194357 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.194321    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:01.194895 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.194391    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:01.194895 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.194413    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:01.212070 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.212028    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 13:03:01.212155 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.212066    2574 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 13:03:01.212517 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.212501    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 13:03:01.226525 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.226505    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-76hdh\""
Oct 30 13:03:01.226677 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.226664    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 13:03:01.226726 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.226686    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hcn4\""
Oct 30 13:03:01.226726 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:01.226710    2574 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzck4\""
Oct 30 13:03:03.972950 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:03.972903    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:03.972950 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:03.972956    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:03.973531 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:03:03.973051    2574 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Oct 30 13:03:03.973531 ip-10-0-136-36 kubenswrapper[2574]: E1030 13:03:03.973114    2574 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs podName:ebb46e24-3980-4909-83fb-bf22d54b5031 nodeName:}" failed. No retries permitted until 2026-10-30 13:03:35.973099459 +0000 UTC m=+66.218953403 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs") pod "network-metrics-daemon-66rk6" (UID: "ebb46e24-3980-4909-83fb-bf22d54b5031") : secret "metrics-daemon-secret" not found
Oct 30 13:03:03.976884 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:03.976854    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:04.007258 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:04.007223    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:04.074034 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:04.073996    2574 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:04.076453 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:04.076421    2574 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:04.282908 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:04.282814    2574 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:04.449703 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:04.449668    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j4n8s"]
Oct 30 13:03:04.453193 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:03:04.453169    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37b49c3d_d533_4d1f_898b_2c2fa83e2126.slice/crio-779fcf3815f06943978ce882994343af8c4a6970583959f0234976ccfa8896df WatchSource:0}: Error finding container 779fcf3815f06943978ce882994343af8c4a6970583959f0234976ccfa8896df: Status 404 returned error can't find the container with id 779fcf3815f06943978ce882994343af8c4a6970583959f0234976ccfa8896df
Oct 30 13:03:04.480575 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:04.480546    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j4n8s" event={"ID":"37b49c3d-d533-4d1f-898b-2c2fa83e2126","Type":"ContainerStarted","Data":"779fcf3815f06943978ce882994343af8c4a6970583959f0234976ccfa8896df"}
Oct 30 13:03:04.578952 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:04.578868    2574 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8542k"]
Oct 30 13:03:04.582515 ip-10-0-136-36 kubenswrapper[2574]: W1030 13:03:04.582484    2574 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28913fb0_2984_4375_8fe0_e15682bde559.slice/crio-2e5980ffe5146426ded41513646ff68358c379794b4e8373da11ce45b2a26474 WatchSource:0}: Error finding container 2e5980ffe5146426ded41513646ff68358c379794b4e8373da11ce45b2a26474: Status 404 returned error can't find the container with id 2e5980ffe5146426ded41513646ff68358c379794b4e8373da11ce45b2a26474
Oct 30 13:03:05.485010 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:05.484932    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8542k" event={"ID":"28913fb0-2984-4375-8fe0-e15682bde559","Type":"ContainerStarted","Data":"2e5980ffe5146426ded41513646ff68358c379794b4e8373da11ce45b2a26474"}
Oct 30 13:03:08.494245 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:08.494201    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8542k" event={"ID":"28913fb0-2984-4375-8fe0-e15682bde559","Type":"ContainerStarted","Data":"b0c9f09763fb84a2502f512da4071bf29be0a567bd4e349acad6ce5f9105f135"}
Oct 30 13:03:08.494733 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:08.494340    2574 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:08.535931 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:08.535867    2574 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8542k" podStartSLOduration=35.268391364 podStartE2EDuration="38.535849099s" totalImagesPullingTime="3.267457735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:02:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:08.535217987 +0000 UTC m=+38.781071933" watchObservedRunningTime="2026-10-30 13:03:08.535849099 +0000 UTC m=+38.781703074"
Oct 30 13:03:09.498323 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:09.498294    2574 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j4n8s" event={"ID":"37b49c3d-d533-4d1f-898b-2c2fa83e2126","Type":"ContainerStarted","Data":"e779cada1f1147fe2b2d2a05747c3a91dd151f982f2e48cc78d464a0394df1a7"}
Oct 30 13:03:09.682128 ip-10-0-136-36 kubenswrapper[2574]: I1030 13:03:09.682035    2574 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:03:09.682206 ip-10-0-136-36 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 13:03:09.694037 ip-10-0-136-36 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 13:03:09.694211 ip-10-0-136-36 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 13:03:09.694260 ip-10-0-136-36 systemd[1]: kubelet.service: Consumed 2.144s CPU time.
Oct 30 13:03:09.709160 ip-10-0-136-36 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 13:03:09.803724 ip-10-0-136-36 kubenswrapper[4291]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:03:09.803724 ip-10-0-136-36 kubenswrapper[4291]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 13:03:09.803724 ip-10-0-136-36 kubenswrapper[4291]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:03:09.803724 ip-10-0-136-36 kubenswrapper[4291]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 13:03:09.805857 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805776    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.805857 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805857    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805862    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805866    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805869    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805872    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805875    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805878    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805880    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805884    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805886    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805889    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805892    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805895    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805901    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805905    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805907    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805910    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805913    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805915    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.805932 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805918    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805921    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805923    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805927    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805931    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805935    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805937    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805940    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805943    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805946    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805949    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805951    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805954    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805957    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805959    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805962    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805965    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805967    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805970    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.806491 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805972    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805974    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805977    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805980    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805982    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805985    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805987    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805990    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805993    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805995    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.805998    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806000    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806004    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806007    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806009    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806012    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806014    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806017    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806019    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.806996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806022    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806025    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806027    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806030    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806032    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806035    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806038    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806041    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806043    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806046    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806048    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806051    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806053    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806056    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806058    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806060    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806063    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806066    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806068    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806070    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.807468 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806073    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806077    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806080    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806083    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806085    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806093    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806096    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806099    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806102    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806105    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806108    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806111    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806113    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806116    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806119    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806122    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806124    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806127    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806129    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.807980 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806132    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806135    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806137    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806139    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806142    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806145    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806148    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806151    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806153    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806939    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806947    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806950    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806953    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806956    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806960    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806966    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806969    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806973    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806979    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.808465 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806982    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806985    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806987    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806990    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806993    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806995    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.806998    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807001    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807003    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807006    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807009    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807012    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807014    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807017    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807020    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807022    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807025    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807027    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807030    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.808948 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807033    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807035    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807038    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807040    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807043    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807046    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807048    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807051    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807053    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807056    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807060    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807063    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807066    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807069    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807072    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807074    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807077    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807079    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807082    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807085    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.809425 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807087    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807090    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807092    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807095    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807097    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807100    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807102    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807105    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807108    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807110    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807113    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807115    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807118    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807120    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807124    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807128    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807131    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807135    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807138    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.809995 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807141    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807144    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807148    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807151    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807154    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807157    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807159    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807162    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807164    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807167    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807170    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807173    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807175    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807178    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807180    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807183    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807185    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807187    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807190    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807192    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.810505 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807195    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807198    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807201    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807204    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807206    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807209    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807212    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807214    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807217    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807219    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807222    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807225    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807227    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807230    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807232    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807235    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807238    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807242    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807788    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807795    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.811025 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807798    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807801    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807804    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807807    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807810    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807812    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807815    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807818    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807820    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807823    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807826    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807828    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807831    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807833    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807836    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807839    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807842    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807844    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807847    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807849    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.811523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807852    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807855    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807857    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807860    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807863    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807865    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807868    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807871    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807873    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807877    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807879    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807882    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807886    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807889    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807892    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807894    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807897    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807900    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807902    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.812058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807905    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807907    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807910    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807912    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807915    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807918    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807920    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807923    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807928    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807930    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807933    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807936    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807939    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807942    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807944    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807947    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807950    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807952    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807955    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.812528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807960    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807963    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807965    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807968    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807971    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807973    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807976    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807979    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807981    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807984    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807987    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807990    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807992    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807995    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.807998    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808000    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808003    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808005    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808008    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808010    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.813011 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808013    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808016    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808019    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808022    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808024    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808027    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808030    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808033    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808037    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808042    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808045    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808048    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808052    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808055    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808058    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808060    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808063    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808066    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808069    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.813756 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808071    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808074    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808076    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808079    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808081    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808084    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808086    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808188    4291 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808199    4291 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808207    4291 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808212    4291 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808217    4291 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808220    4291 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808225    4291 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808234    4291 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808237    4291 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808240    4291 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808244    4291 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808247    4291 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808250    4291 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808253    4291 flags.go:64] FLAG: --cgroup-root=""
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808256    4291 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808259    4291 flags.go:64] FLAG: --client-ca-file=""
Oct 30 13:03:09.814525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808262    4291 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808265    4291 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808271    4291 flags.go:64] FLAG: --cluster-domain=""
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808274    4291 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808277    4291 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808280    4291 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808283    4291 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808287    4291 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808290    4291 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808294    4291 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808298    4291 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808301    4291 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808304    4291 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808307    4291 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808310    4291 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808313    4291 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808317    4291 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808320    4291 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808323    4291 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808326    4291 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808329    4291 flags.go:64] FLAG: --enable-server="true"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808332    4291 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808336    4291 flags.go:64] FLAG: --event-burst="100"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808341    4291 flags.go:64] FLAG: --event-qps="50"
Oct 30 13:03:09.815165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808344    4291 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808347    4291 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808350    4291 flags.go:64] FLAG: --eviction-hard=""
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808354    4291 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808357    4291 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808361    4291 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808364    4291 flags.go:64] FLAG: --eviction-soft=""
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808367    4291 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808370    4291 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808373    4291 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808378    4291 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808381    4291 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808384    4291 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808387    4291 flags.go:64] FLAG: --feature-gates=""
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808391    4291 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808394    4291 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808397    4291 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808400    4291 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808404    4291 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808407    4291 flags.go:64] FLAG: --help="false"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808410    4291 flags.go:64] FLAG: --hostname-override="ip-10-0-136-36.ec2.internal"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808413    4291 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808416    4291 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 13:03:09.815824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808419    4291 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808422    4291 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808425    4291 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808429    4291 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808432    4291 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808435    4291 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808438    4291 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808441    4291 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808445    4291 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808448    4291 flags.go:64] FLAG: --kube-reserved=""
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808451    4291 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808454    4291 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808457    4291 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808460    4291 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808463    4291 flags.go:64] FLAG: --lock-file=""
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808466    4291 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808469    4291 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808472    4291 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808478    4291 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808482    4291 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808485    4291 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808488    4291 flags.go:64] FLAG: --logging-format="text"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808491    4291 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808494    4291 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 13:03:09.816381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808497    4291 flags.go:64] FLAG: --manifest-url=""
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808501    4291 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808505    4291 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808509    4291 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808513    4291 flags.go:64] FLAG: --max-pods="110"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808516    4291 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808519    4291 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808522    4291 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808525    4291 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808528    4291 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808531    4291 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808534    4291 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808541    4291 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808544    4291 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808547    4291 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808551    4291 flags.go:64] FLAG: --pod-cidr=""
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808556    4291 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808559    4291 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808562    4291 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808565    4291 flags.go:64] FLAG: --port="10250"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808568    4291 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808571    4291 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0cdb13b08d46689ee"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808574    4291 flags.go:64] FLAG: --qos-reserved=""
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808577    4291 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808580    4291 flags.go:64] FLAG: --register-node="true"
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808583    4291 flags.go:64] FLAG: --register-with-taints=""
Oct 30 13:03:09.817022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808587    4291 flags.go:64] FLAG: --registry-burst="10"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808592    4291 flags.go:64] FLAG: --registry-qps="5"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808595    4291 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808598    4291 flags.go:64] FLAG: --reserved-memory=""
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808602    4291 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808605    4291 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808608    4291 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808611    4291 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808614    4291 flags.go:64] FLAG: --runonce="false"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808618    4291 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808621    4291 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808624    4291 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808627    4291 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808643    4291 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808647    4291 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808650    4291 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808653    4291 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808656    4291 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808659    4291 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808662    4291 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808665    4291 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808668    4291 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808672    4291 flags.go:64] FLAG: --system-cgroups=""
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808676    4291 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808681    4291 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 13:03:09.817688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808684    4291 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808687    4291 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808691    4291 flags.go:64] FLAG: --tls-min-version=""
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808694    4291 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808697    4291 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808700    4291 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808703    4291 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808706    4291 flags.go:64] FLAG: --v="2"
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808711    4291 flags.go:64] FLAG: --version="false"
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808716    4291 flags.go:64] FLAG: --vmodule=""
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808720    4291 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.808723    4291 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808851    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808854    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808857    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808859    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808862    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808864    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808868    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808871    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808873    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808876    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.818306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808879    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808882    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808884    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808887    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808890    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808892    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808895    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808900    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808904    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808907    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808910    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808913    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808916    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808918    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808921    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808923    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808926    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808928    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808933    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.818873 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808935    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808938    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808941    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808943    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808946    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808949    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808952    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808954    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808957    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808960    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808963    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808965    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808968    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808970    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808973    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808976    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808978    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808981    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808984    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808986    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.819326 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808990    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808993    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808996    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.808998    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809001    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809003    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809006    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809008    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809011    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809013    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809016    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809019    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809022    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809025    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809027    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809030    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809032    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809035    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809038    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809040    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.819997 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809043    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809045    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809048    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809051    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809053    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809057    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809061    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809064    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809067    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809070    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809073    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809075    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809080    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809082    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809085    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809087    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809090    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809092    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809095    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.820630 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809097    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809100    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809102    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809105    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809109    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809112    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809114    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809117    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809119    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809122    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809125    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809127    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809130    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809132    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809135    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809137    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809140    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.821162 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809152    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.809179    4291 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809214    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809219    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809223    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809226    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809228    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809231    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809235    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809238    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809240    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809243    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809245    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809248    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809250    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.821753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809253    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809255    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809258    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809261    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809263    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809267    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809270    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809272    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809275    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809278    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809280    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809283    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809286    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809288    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809291    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809294    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809296    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809299    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809302    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809305    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.822125 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809307    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809311    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809313    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809316    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809318    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809322    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809325    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809327    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809330    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809332    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809335    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809338    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809340    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809343    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809345    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809348    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809350    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809354    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809357    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809359    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.822618 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809362    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809364    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809367    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809370    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809373    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809377    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809380    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809383    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809385    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809388    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809391    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809394    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809396    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809399    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809401    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809404    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809407    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809410    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809413    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.823340 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809415    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809418    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809421    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809423    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809426    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809428    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809431    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809434    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809436    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809439    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809443    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809446    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809448    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809451    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809453    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809456    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809459    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809461    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809464    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.823831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809466    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809469    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809471    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809474    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809476    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809479    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809482    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809485    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809487    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809490    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809492    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809496    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809499    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809502    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.809504    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.815084    4291 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.815103    4291 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.815126    4291 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.815131    4291 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815192    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.824338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815198    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815201    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815205    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815207    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815210    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815213    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815216    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815219    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815222    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815224    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815227    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815230    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815233    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815235    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815238    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815241    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815244    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815246    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815249    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815251    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.824836 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815255    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815258    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815260    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815263    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815266    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815268    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815271    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815275    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815277    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815280    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815283    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815285    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815288    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815291    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815293    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815296    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815299    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815302    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815304    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815307    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.825332 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815310    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815312    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815315    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815317    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815320    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815322    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815325    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815328    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815330    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815333    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815335    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815338    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815341    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815344    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815347    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815349    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815352    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815355    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815358    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815361    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.825885 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815363    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815366    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815368    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815371    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815374    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815377    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815379    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815382    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815384    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815387    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815389    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815392    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815394    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815397    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815399    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815402    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815404    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815407    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815416    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815419    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.826387 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815421    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815424    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815427    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815430    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815432    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815435    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815438    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815441    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815445    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815450    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815453    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815456    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815459    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815461    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815464    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815466    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815469    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815471    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.826955 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815474    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815478    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815482    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815485    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815488    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815490    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815493    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.815522    4291 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815681    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815687    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815690    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815693    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815696    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815705    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815708    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815710    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.827377 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815714    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815716    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815719    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815722    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815725    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815727    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815730    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815733    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815735    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815738    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815741    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815743    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815746    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815749    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815752    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815754    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815757    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815759    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815762    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.827775 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815764    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815767    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815769    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815772    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815775    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815778    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815781    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815784    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815790    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815792    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815795    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815803    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815806    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815809    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815812    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815814    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815817    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815820    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815822    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.828222 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815825    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815828    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815830    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815833    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815836    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815839    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815841    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815844    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815847    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815850    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815852    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815855    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815857    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815860    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815862    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815865    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815867    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815870    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815872    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815875    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.828702 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815877    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815882    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815885    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815887    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815890    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815898    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815901    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815904    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815906    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815909    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815912    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815914    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815917    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815920    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815922    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815925    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815928    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815930    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815933    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815935    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.829179 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815938    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815940    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815943    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815945    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815948    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815951    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815953    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815955    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815958    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815960    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815963    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815965    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815968    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815972    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815975    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815977    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815981    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815985    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815995    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.829677 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.815998    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.816023    4291 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816060    4291 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816065    4291 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816068    4291 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816070    4291 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816073    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816076    4291 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816078    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816081    4291 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816084    4291 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816086    4291 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816089    4291 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816091    4291 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816094    4291 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816097    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 13:03:09.830136 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816099    4291 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816102    4291 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816105    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816107    4291 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816110    4291 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816112    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816115    4291 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816117    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816120    4291 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816123    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816128    4291 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816130    4291 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816133    4291 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816135    4291 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816138    4291 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816140    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816150    4291 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816153    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816156    4291 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816158    4291 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 13:03:09.830508 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816161    4291 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816164    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816166    4291 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816169    4291 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816172    4291 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816174    4291 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816177    4291 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816180    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816182    4291 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816185    4291 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816187    4291 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816190    4291 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816192    4291 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816195    4291 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816198    4291 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816200    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816203    4291 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816205    4291 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816208    4291 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816210    4291 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 13:03:09.831098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816213    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816216    4291 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816222    4291 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816225    4291 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816228    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816231    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816234    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816237    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816241    4291 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816249    4291 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816252    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816254    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816257    4291 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816260    4291 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816263    4291 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816266    4291 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816269    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816271    4291 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816274    4291 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 13:03:09.831724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816276    4291 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816279    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816282    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816284    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816287    4291 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816290    4291 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816292    4291 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816295    4291 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816298    4291 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816300    4291 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816303    4291 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816306    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816308    4291 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816311    4291 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816313    4291 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816319    4291 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816323    4291 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816327    4291 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816334    4291 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 13:03:09.832447 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816339    4291 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816342    4291 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816347    4291 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816349    4291 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816358    4291 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816361    4291 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816365    4291 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816367    4291 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816370    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816372    4291 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816375    4291 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816378    4291 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816380    4291 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:09.816383    4291 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.816464    4291 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.817485    4291 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.817538    4291 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.817729    4291 server.go:1014] "Starting client certificate rotation"
Oct 30 13:03:09.833041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.817863    4291 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.817907    4291 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 12:57:30 +0000 UTC" deadline="2028-06-08 10:46:56.402923655 +0000 UTC"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.817945    4291 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14085h43m46.584980685s"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.818225    4291 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.819442    4291 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.821204    4291 log.go:25] "Validated CRI v1 runtime API"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.822676    4291 log.go:25] "Validated CRI v1 image API"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.823434    4291 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 13:03:09.833518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.824817    4291 fs.go:135] Filesystem UUIDs: map[75903a49-d32c-4be3-8bde-60ac9c3f35c2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 9fb4bca6-2e7a-4eb1-9a77-ce5e69419e63:/dev/nvme0n1p3]
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.824837    4291 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/262a20c7f962dbf5f51a88684f98cad8effd2231a8fabbc0eccb68edbc3e2ce6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/262a20c7f962dbf5f51a88684f98cad8effd2231a8fabbc0eccb68edbc3e2ce6/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2e5980ffe5146426ded41513646ff68358c379794b4e8373da11ce45b2a26474/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2e5980ffe5146426ded41513646ff68358c379794b4e8373da11ce45b2a26474/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e83be51b4dbf0c424325b0a7fd4841789fe2010d268b780e7b692dfd15a27bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e83be51b4dbf0c424325b0a7fd4841789fe2010d268b780e7b692dfd15a27bc/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e968d877a5022c9878e52155f0e8cd982671071ab9d3c304c8b006fd4d16cb7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e968d877a5022c9878e52155f0e8cd982671071ab9d3c304c8b006fd4d16cb7/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/779fcf3815f06943978ce882994343af8c4a6970583959f0234976ccfa8896df/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/779fcf3815f06943978ce882994343af8c4a6970583959f0234976ccfa8896df/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a73423083f2ae4dfca58d389e5d518e9b727b6f4039621c31c186ef8fd09d367/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a73423083f2ae4dfca58d389e5d518e9b727b6f4039621c31c186ef8fd09d367/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1ed60384455e228b7939b788f2e262172466aecbb83bc89f36ed6b455f0a2d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1ed60384455e228b7939b788f2e262172466aecbb83bc89f36ed6b455f0a2d7/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c0789d6ded7a57a3e444a83cdc0b9b998ebb97ca6cc86ddd447deb4ac8fa13c9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c0789d6ded7a57a3e444a83cdc0b9b998ebb97ca6cc86ddd447deb4ac8fa13c9/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c60a10900bae5fc4c2ec5d7a6530b5ce0b239d539e0c7c24f3e09088f6ea01a4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c60a10900bae5fc4c2ec5d7a6530b5ce0b239d539e0c7c24f3e09088f6ea01a4/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d2e1a863f07002790a0c7b8f2479780309a75ae368f87cc80e15c33608b77a50/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d2e1a863f07002790a0c7b8f2479780309a75ae368f87cc80e15c33608b77a50/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/02f4d975-e0f4-456c-a85e-2a1389468f5c/volumes/kubernetes.io~projected/kube-api-access-qjzhw:{mountpoint:/var/lib/kubelet/pods/02f4d975-e0f4-456c-a85e-2a1389468f5c/volumes/kubernetes.io~projected/kube-api-access-qjzhw major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/20cc5462-99f5-410c-97d6-b414507f297a/volumes/kubernetes.io~projected/kube-api-access-6x75s:{mountpoint:/var/lib/kubelet/pods/20cc5462-99f5-410c-97d6-b414507f297a/volumes/kubernetes.io~projected/kube-api-access-6x75s major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/28913fb0-2984-4375-8fe0-e15682bde559/volumes/kubernetes.io~projected/kube-api-access-gnz2d:{mountpoint:/var/lib/kubelet/pods/28913fb0-2984-4375-8fe0-e15682bde559/volumes/kubernetes.io~projected/kube-api-access-gnz2d major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/37b49c3d-d533-4d1f-898b-2c2fa83e2126/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/37b49c3d-d533-4d1f-898b-2c2fa83e2126/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volumes/kubernetes.io~projected/kube-api-access-6l8z4:{mountpoint:/var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volumes/kubernetes.io~projected/kube-api-access-6l8z4 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ec2afa3-085d-4e46-8488-60154d80ec91/volumes/kubernetes.io~projected/kube-api-access-h9pbg:{mountpoint:/var/lib/kubelet/pods/7ec2afa3-085d-4e46-8488-60154d80ec91/volumes/kubernetes.io~projected/kube-api-access-h9pbg major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9bc491f0-f83f-48fa-8f83-92f88f9f9467/volumes/kubernetes.io~projected/kube-api-access-jnjj8:{mountpoint:/var/lib/kubelet/pods/9bc491f0-f83f-48fa-8f83-92f88f9f9467/volumes/kubernetes.io~projected/kube-api-access-jnjj8 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b331a27e-e822-4419-b000-bf767335246a/volumes/kubernetes.io~projected/kube-api-access-k4gz6:{mountpoint:/var/lib/kubelet/pods/b331a27e-e822-4419-b000-bf767335246a/volumes/kubernetes.io~projected/kube-api-access-k4gz6 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~projected/kube-api-access-k5zlw:{mountpoint:/var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~projected/kube-api-access-k5zlw major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da470b83-8d33-4fe6-a4c9-2c3a0b85d060/volumes/kubernetes.io~projected/kube-api-access-6ftjq:{mountpoint:/var/lib/kubelet/pods/da470b83-8d33-4fe6-a4c9-2c3a0b85d060/volumes/kubernetes.io~projected/kube-api-access-6ftjq major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ebb46e24-3980-4909-83fb-bf22d54b5031/volumes/kubernetes.io~projected/kube-api-access-t8ww8:{mountpoint:/var/lib/kubelet/pods/ebb46e24-3980-4909-83fb-bf22d54b5031/volumes/kubernetes.io~projected/kube-api-access-t8ww8 major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/568eec993a294ecf7c0406220a229f2b77398ece3d9eb48deac3cdc0cb9a6ee6/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/e294f37984dc222cf1554b3835877c9ba78d352c4321662ec67a9f4a0688d0dd/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/41ac8f64249954125f5a9c4f454244580015003eb9503cc9ed8683ae9108256c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/7a6d66e4c14a22a714d00b2db8e317fc0a4c4c43afdfef16be2c7b876ad9a147/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/e2b2938ec2de6dae9c3e59205ba9444ecbbaedbb066a2f15d993f1e552dc0e14/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/53e66fddd978317dcfd0906451d83dbd7283f5c201bd3b5fdcc9f281597e8ebc/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3848399ff0836804a4b6a34e9976b1b5361a43c6370fc428172b2af3c52f4009/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/5f66ff077b295571036a235e8d5051982562f3a725c17a2977c2f35d46ed80fc/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/82de9a21b4254678bac88c9c16cc0b6051b1b8c950cd65a08c90a85db63a8f76/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a3362b0eef2880cd6a5bfa99b60d1f9feb5c57ccc37d73f6e017cf7fc095b0fa/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/3c257a2c591565004eb2e9edc1755b97d90bd3ac10d89aa8f32c1da418ce1e36/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-145:{mountpoint:/var/lib/containers/storage/overlay/7d077877d2de19003dc85926be5f292cb5a1485194e2de62d78b8cd18aa1bbfd/merged major:0 minor:145 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/2f164970e9c69de63a9501811121a41659025c6832ec7f0b4127d21ca721eb94/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/d7faf58dd1d1f0b1ee9c99f57878379be7c09099cd99abd7071240510553f6e4/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/ac18b1b691ed776498d25877236f7632545b2d3366ad47323fb5b3627e949a39/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fe9813f59598baa466ec220f3667dca1aa8b079afe01f744b44f60dd6a5bf1ee/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/d8b72f8ec4965a0bd6eff164614e6fc999c4209521ea86e9e0c85d41ac4972e5/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f2f504a84d456c446ab58bfe6c6ed00247e55dac2c24e7623f77c1672a82385f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b0c032a4ee9a429b577c63c18a4009b4c8bd7adcb7b2bb302c50a2c68ba0e4a6/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/4396f9ed05e6fe91e9948028865893443792d6b1239247b29e9f413db6607650/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/dd74606298cb2041641544b0c3db8af0be54b0a7a522847642ff62a0f91dde89/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/97e42c75444f78bedbf4337dc63e3dca8afc9aac03392e507dcb3845390916ce/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/6a0403a1bbf0c075f6f4756f09719fea4afeab3e2d5a43e2071a721b031469d0/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/2317d4b8c6507b09ee40e707f8eed14c27a9daf63ee551e0ee0156d0c459a0d5/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/80333697e1a1cf7c30df9ce034aee0709839570f191ea39ee262513f53f4b9d3/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/98c5acb00da368ec303644e22bcf6d6ca020c82267f5965ae6811648f0657e06/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ec110fe460611e27d6ca5a3a89ae7fbc791c6e55635785f9ec0eceba04b3a994/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/c24ac32d2b4ef3bbbf12b84520260b726d61f1964940b55410b3f4868b0f42e8/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/c8a4b844ed4f31248116937ee53ea8c8d140e603ab7591857e13654d6cc681f5/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/c23143a3545028c4101bdfb98c78bf7d5426f65b770174b961dcc4eae71b54ae/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-83:{mountpoint:/var/lib/containers/storage/overlay/3e9a466d0dc24c2a6340d8487801470d3baf3fdd4bbb51903db099387f5a556e/merged major:0 minor:83 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/2644b74b26504a9c933b718ca8c70fbda69d393b0f088205677837a451e9c008/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/7ba1d976ecb6481e9485bff45bf378ace82ed94043931552711438b89a368a12/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/c48208e404270f2859434d052c4250b68685fbac3b33a279e1dba29b02fe356c/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/d1fc3c57e6c55613ac40c3c1993c321adb8716c445d91cf3dfd69e34783d4b79/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.831077    4291 manager.go:217] Machine: {Timestamp:2026-10-30 13:03:09.830570705 +0000 UTC m=+0.090808930 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102361 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20e04913e21e000a00bfbb5e67092a SystemUUID:ec20e049-13e2-1e00-0a00-bfbb5e67092a BootID:fad49515-fd18-475a-905c-6cf100944b63 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e968d877a5022c9878e52155f0e8cd982671071ab9d3c304c8b006fd4d16cb7/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7ec2afa3-085d-4e46-8488-60154d80ec91/volumes/kubernetes.io~projected/kube-api-access-h9pbg DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c0789d6ded7a57a3e444a83cdc0b9b998ebb97ca6cc86ddd447deb4ac8fa13c9/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-83 DeviceMajor:0 DeviceMinor:83 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d2e1a863f07002790a0c7b8f2479780309a75ae368f87cc80e15c33608b77a50/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/02f4d975-e0f4-456c-a85e-2a1389468f5c/volumes/kubernetes.io~projected/kube-api-access-qjzhw DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/20cc5462-99f5-410c-97d6-b414507f297a/volumes/kubernetes.io~projected/kube-api-access-6x75s DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a73423083f2ae4dfca58d389e5d518e9b727b6f4039621c31c186ef8fd09d367/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e83be51b4dbf0c424325b0a7fd4841789fe2010d268b780e7b692dfd15a27bc/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/262a20c7f962dbf5f51a88684f98cad8effd2231a8fabbc0eccb68edbc3e2ce6/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1ed60384455e228b7939b788f2e262172466aecbb83bc89f36ed6b455f0a2d7/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/779fcf3815f06943978ce882994343af8c4a6970583959f0234976ccfa8896df/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ebb46e24-3980-4909-83fb-bf22d54b5031/volumes/kubernetes.io~projected/kube-api-access-t8ww8 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2e5980ffe5146426ded41513646ff68358c379794b4e8373da11ce45b2a26474/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/b331a27e-e822-4419-b000-bf767335246a/volumes/kubernetes.io~projected/kube-api-access-k4gz6 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/da470b83-8d33-4fe6-a4c9-2c3a0b85d060/volumes/kubernetes.io~projected/kube-api-access-6ftjq DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-145 DeviceMajor:0 DeviceMinor:145 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/28913fb0-2984-4375-8fe0-e15682bde559/volumes/kubernetes.io~projected/kube-api-access-gnz2d DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c56d84ee-39fa-48a9-9426-60b730cfaccc/volumes/kubernetes.io~projected/kube-api-access-k5zlw DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/37b49c3d-d533-4d1f-898b-2c2fa83e2126/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volumes/kubernetes.io~projected/kube-api-access-6l8z4 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/9bc491f0-f83f-48fa-8f83-92f88f9f9467/volumes/kubernetes.io~projected/kube-api-access-jnjj8 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c60a10900bae5fc4c2ec5d7a6530b5ce0b239d539e0c7c24f3e09088f6ea01a4/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/723b2da3-c191-420c-b6d1-7f8a4a9806cd/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:2e5980ffe514642 MacAddress:ea:d7:20:3c:8d:5d Speed:10000 Mtu:8901} {Name:779fcf3815f0694 MacAddress:42:9b:f6:fa:0f:59 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:de:d0:d7:6f Speed:0 Mtu:9001} {Name:br-int MacAddress:f6:eb:b9:72:b5:8e Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:de:d0:d7:6f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:9a:d8:81:30:8d:ed Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:46:9d:9c:bf:e6:9b Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.831301    4291 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.831343    4291 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.832132    4291 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.832158    4291 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-36.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.832527    4291 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.832541    4291 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.832554    4291 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.832585    4291 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.832827    4291 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.833198    4291 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.833325    4291 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.833363    4291 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 13:03:09.833784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.833376    4291 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 13:03:09.834327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.833387    4291 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 13:03:09.834327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.833396    4291 kubelet.go:405] "Adding apiserver pod source"
Oct 30 13:03:09.834327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.833404    4291 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 13:03:09.834327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.834267    4291 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 13:03:09.834327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.834281    4291 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:03:09.834823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.834806    4291 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 13:03:09.836786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.836772    4291 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 13:03:09.836786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.836786    4291 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 13:03:09.836894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.836884    4291 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 13:03:09.837080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837069    4291 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 13:03:09.837110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837099    4291 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 13:03:09.837200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837191    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 13:03:09.837233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837208    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 13:03:09.837233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837215    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 13:03:09.837233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837224    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 13:03:09.837233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837233    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837241    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837250    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837259    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837267    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837274    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837289    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837299    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837324    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 13:03:09.837331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.837330    4291 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 13:03:09.840552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.840534    4291 server.go:1263] "Started kubelet"
Oct 30 13:03:09.840689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.840609    4291 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 13:03:09.840749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.840652    4291 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 13:03:09.840749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.840735    4291 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 13:03:09.841131 ip-10-0-136-36 systemd[1]: Started Kubernetes Kubelet.
Oct 30 13:03:09.841450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.841215    4291 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 13:03:09.842717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.842624    4291 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 13:03:09.843739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.843422    4291 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 13:03:09.843739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.843454    4291 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 13:03:09.843739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.843544    4291 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 13:03:09.843739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.843553    4291 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 13:03:09.844619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.844403    4291 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 13:03:09.844619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.843518    4291 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 12:57:32 +0000 UTC" deadline="2028-04-10 16:18:02.322385612 +0000 UTC"
Oct 30 13:03:09.844619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.844437    4291 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12675h14m52.477952924s"
Oct 30 13:03:09.844961 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:09.843570    4291 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-36.ec2.internal\" not found"
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846657    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c56d84ee-39fa-48a9-9426-60b730cfaccc" volumeName="kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-tuned" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846764    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ebb46e24-3980-4909-83fb-bf22d54b5031" volumeName="kubernetes.io/projected/ebb46e24-3980-4909-83fb-bf22d54b5031-kube-api-access-t8ww8" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846788    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="723b2da3-c191-420c-b6d1-7f8a4a9806cd" volumeName="kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-config" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846805    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc491f0-f83f-48fa-8f83-92f88f9f9467" volumeName="kubernetes.io/projected/9bc491f0-f83f-48fa-8f83-92f88f9f9467-kube-api-access-jnjj8" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846825    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b331a27e-e822-4419-b000-bf767335246a" volumeName="kubernetes.io/configmap/b331a27e-e822-4419-b000-bf767335246a-iptables-alerter-script" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846839    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b331a27e-e822-4419-b000-bf767335246a" volumeName="kubernetes.io/projected/b331a27e-e822-4419-b000-bf767335246a-kube-api-access-k4gz6" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846854    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c56d84ee-39fa-48a9-9426-60b730cfaccc" volumeName="kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-tmp" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846874    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="723b2da3-c191-420c-b6d1-7f8a4a9806cd" volumeName="kubernetes.io/projected/723b2da3-c191-420c-b6d1-7f8a4a9806cd-kube-api-access-6l8z4" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846892    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02f4d975-e0f4-456c-a85e-2a1389468f5c" volumeName="kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-binary-copy" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846911    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02f4d975-e0f4-456c-a85e-2a1389468f5c" volumeName="kubernetes.io/projected/02f4d975-e0f4-456c-a85e-2a1389468f5c-kube-api-access-qjzhw" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846925    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3" volumeName="kubernetes.io/secret/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-agent-certs" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846945    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="37b49c3d-d533-4d1f-898b-2c2fa83e2126" volumeName="kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846959    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec2afa3-085d-4e46-8488-60154d80ec91" volumeName="kubernetes.io/projected/7ec2afa3-085d-4e46-8488-60154d80ec91-kube-api-access-h9pbg" seLinuxMountContext=""
Oct 30 13:03:09.847033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.846974    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc491f0-f83f-48fa-8f83-92f88f9f9467" volumeName="kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cni-binary-copy" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847742    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9bc491f0-f83f-48fa-8f83-92f88f9f9467" volumeName="kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-daemon-config" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847781    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02f4d975-e0f4-456c-a85e-2a1389468f5c" volumeName="kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847802    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cc5462-99f5-410c-97d6-b414507f297a" volumeName="kubernetes.io/empty-dir/20cc5462-99f5-410c-97d6-b414507f297a-tmp-dir" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847818    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="28913fb0-2984-4375-8fe0-e15682bde559" volumeName="kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847845    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="723b2da3-c191-420c-b6d1-7f8a4a9806cd" volumeName="kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-env-overrides" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847879    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="723b2da3-c191-420c-b6d1-7f8a4a9806cd" volumeName="kubernetes.io/secret/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847893    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ec2afa3-085d-4e46-8488-60154d80ec91" volumeName="kubernetes.io/configmap/7ec2afa3-085d-4e46-8488-60154d80ec91-serviceca" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847911    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c56d84ee-39fa-48a9-9426-60b730cfaccc" volumeName="kubernetes.io/projected/c56d84ee-39fa-48a9-9426-60b730cfaccc-kube-api-access-k5zlw" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847924    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da470b83-8d33-4fe6-a4c9-2c3a0b85d060" volumeName="kubernetes.io/projected/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kube-api-access-6ftjq" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847943    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="02f4d975-e0f4-456c-a85e-2a1389468f5c" volumeName="kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847957    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3" volumeName="kubernetes.io/configmap/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-konnectivity-ca" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847972    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="20cc5462-99f5-410c-97d6-b414507f297a" volumeName="kubernetes.io/projected/20cc5462-99f5-410c-97d6-b414507f297a-kube-api-access-6x75s" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.847991    4291 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="723b2da3-c191-420c-b6d1-7f8a4a9806cd" volumeName="kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.848005    4291 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.848015    4291 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.848837    4291 factory.go:55] Registering systemd factory
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.848855    4291 factory.go:223] Registration of the systemd container factory successfully
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.849245    4291 factory.go:153] Registering CRI-O factory
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.849259    4291 factory.go:223] Registration of the crio container factory successfully
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.849304    4291 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.849319    4291 factory.go:103] Registering Raw factory
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.849337    4291 manager.go:1199] Started watching for new ooms in manager
Oct 30 13:03:09.851031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.849618    4291 manager.go:319] Starting recovery of all containers
Oct 30 13:03:09.868583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.868536    4291 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 13:03:09.870413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.870363    4291 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 13:03:09.870413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.870416    4291 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 13:03:09.870599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.870444    4291 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 13:03:09.870599 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:09.870492    4291 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 13:03:09.893086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.893046    4291 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:09.895067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.895043    4291 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 13:03:09.895187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.895132    4291 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:09.896266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.896232    4291 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="6618dd61e2ba7ada7d935c42029111d5bc0cceeac44ce7fd8ee786de1956c6ab" exitCode=0
Oct 30 13:03:09.896266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.896266    4291 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="bc6927950a222704000c9289725521955ae6cd84ec3744a57edc10bce522fcb8" exitCode=0
Oct 30 13:03:09.896413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.896276    4291 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="3827684666aeabfe3bf39e3fb8c08b4210fac67df9f172aac5740c6890498d25" exitCode=0
Oct 30 13:03:09.896413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.896289    4291 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="62678e4f2255cfcc5a3c37362143e49cc8b330dc970973216a9a1047ccb8a62a" exitCode=0
Oct 30 13:03:09.896413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.896300    4291 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="f13ae91d0c2caaa3e879c1994a5ecb46a1c445b40be2e5edbfecbcdd60d0668b" exitCode=0
Oct 30 13:03:09.896413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.896309    4291 generic.go:356] "Generic (PLEG): container finished" podID="02f4d975-e0f4-456c-a85e-2a1389468f5c" containerID="b5706ee4fb747b6b343856e88e339569ed1a417c80b7a90090e3a65a6af80cf3" exitCode=0
Oct 30 13:03:09.897899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.897868    4291 generic.go:356] "Generic (PLEG): container finished" podID="7120be761a1303883792456d28b86647" containerID="48c341ed88af414ef642db376cdb62c95960ebb5efdb043b814a6b0a779ca789" exitCode=0
Oct 30 13:03:09.907434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.907403    4291 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:09.907589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.907569    4291 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 13:03:09.910234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.910204    4291 manager.go:324] Recovery completed
Oct 30 13:03:09.924230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924209    4291 cpu_manager.go:226] "Starting" policy="none"
Oct 30 13:03:09.924230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924224    4291 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 13:03:09.924358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924241    4291 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 13:03:09.924358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924353    4291 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 13:03:09.924426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924363    4291 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 13:03:09.924426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924380    4291 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 13:03:09.924426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924386    4291 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 13:03:09.924426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924391    4291 policy_none.go:50] "Start"
Oct 30 13:03:09.924426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924398    4291 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 13:03:09.924426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924406    4291 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 13:03:09.924580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924518    4291 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 13:03:09.924580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924524    4291 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 13:03:09.924580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.924530    4291 policy_none.go:44] "Start"
Oct 30 13:03:09.928375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.928349    4291 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 13:03:09.930915 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.930902    4291 manager.go:341] "Starting Device Plugin manager"
Oct 30 13:03:09.931381 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:09.930926    4291 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 13:03:09.931441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.931384    4291 server.go:87] "Starting device plugin registration server"
Oct 30 13:03:09.931707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.931696    4291 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 13:03:09.931754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.931708    4291 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 13:03:09.931846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.931824    4291 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 13:03:09.931981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.931932    4291 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 13:03:09.932038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.931984    4291 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 13:03:09.932038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.931991    4291 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 13:03:09.932210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.932116    4291 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T13:03:09.931970678Z","UUID":"0fed6421-886e-433b-a7cb-6e3d7ff96828","Handler":null,"Name":"","Endpoint":""}
Oct 30 13:03:09.933128 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:09.933110    4291 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 13:03:09.933512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.933496    4291 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 13:03:09.933589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.933519    4291 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 13:03:09.970687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.970609    4291 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"]
Oct 30 13:03:09.971182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.971140    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal" event={"ID":"7120be761a1303883792456d28b86647","Type":"ContainerStarted","Data":"72506b471189f74f189e13c340d9dad43e26f29e8df7ee462350e67ad7446510"}
Oct 30 13:03:09.971290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.971197    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal" event={"ID":"7120be761a1303883792456d28b86647","Type":"ContainerDied","Data":"48c341ed88af414ef642db376cdb62c95960ebb5efdb043b814a6b0a779ca789"}
Oct 30 13:03:09.971290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.971215    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal" event={"ID":"7120be761a1303883792456d28b86647","Type":"ContainerStarted","Data":"683cc1622b54266c92dfe8f5cf632eedf67b3791499953f15ebedf6aded0aef5"}
Oct 30 13:03:09.971290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.971227    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal" event={"ID":"3da04c9aa773780d1e2c9af45a694726","Type":"ContainerStarted","Data":"ab2ae0fc353b172945c7fc9135be4b100df0f4e504995491127d34c315b7ce62"}
Oct 30 13:03:09.971290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.971240    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal" event={"ID":"3da04c9aa773780d1e2c9af45a694726","Type":"ContainerStarted","Data":"9401351cff6a8995641e21b03b640f95159a3f0c7b45cd89aa7395564a1a5b35"}
Oct 30 13:03:09.971290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.971281    4291 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:09.971490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.971347    4291 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:09.993086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.993053    4291 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:09.993206 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:09.993108    4291 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:09.993464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:09.993451    4291 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:09.993522 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:09.993473    4291 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.032782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.032749    4291 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 13:03:10.034119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.034098    4291 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientMemory"
Oct 30 13:03:10.034240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.034134    4291 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasNoDiskPressure"
Oct 30 13:03:10.034240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.034145    4291 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-36.ec2.internal" event="NodeHasSufficientPID"
Oct 30 13:03:10.034240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.034174    4291 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.069034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.069004    4291 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.069171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.069081    4291 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.095923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.095894    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3da04c9aa773780d1e2c9af45a694726-config\") pod \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" (UID: \"3da04c9aa773780d1e2c9af45a694726\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.095923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.095925    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.096130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.095944    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.196311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.196224    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.196311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.196260    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.196311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.196278    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3da04c9aa773780d1e2c9af45a694726-config\") pod \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" (UID: \"3da04c9aa773780d1e2c9af45a694726\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.196512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.196324    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3da04c9aa773780d1e2c9af45a694726-config\") pod \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" (UID: \"3da04c9aa773780d1e2c9af45a694726\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.196512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.196341    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.196512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.196342    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7120be761a1303883792456d28b86647-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" (UID: \"7120be761a1303883792456d28b86647\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.834476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.834442    4291 apiserver.go:52] "Watching apiserver"
Oct 30 13:03:10.873421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.873395    4291 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 13:03:10.873799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.873781    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg","openshift-image-registry/node-ca-gr5nl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal","openshift-network-operator/iptables-alerter-nlmnl","kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal","openshift-multus/multus-lz7gh","openshift-multus/network-metrics-daemon-66rk6","kube-system/konnectivity-agent-xnlf9","openshift-cluster-node-tuning-operator/tuned-k52kk","openshift-multus/multus-additional-cni-plugins-c2nts","openshift-ovn-kubernetes/ovnkube-node-28lqx","kube-system/global-pull-secret-syncer-j4n8s","openshift-dns/node-resolver-5sstj","openshift-network-diagnostics/network-check-target-8542k"]
Oct 30 13:03:10.874181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.874163    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:10.889157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889126    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzck4\""
Oct 30 13:03:10.889274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889170    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-sxrfm\""
Oct 30 13:03:10.889274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889188    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 13:03:10.889274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889222    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 13:03:10.889441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889390    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.889441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889418    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 13:03:10.889441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889428    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 13:03:10.889441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889423    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.889569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889536    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.889569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889545    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.889643 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889589    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 13:03:10.889643 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889614    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.889715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889673    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.889767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889755    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9hcn4\""
Oct 30 13:03:10.889878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889863    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m8jrt\""
Oct 30 13:03:10.889909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889880    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 13:03:10.889940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.889925    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.901926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.901909    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.902317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902302    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.902364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902346    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.902414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902387    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-rrlh9\""
Oct 30 13:03:10.902456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902427    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 13:03:10.902507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902476    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-mxh59\""
Oct 30 13:03:10.902789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902767    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8w9b7\""
Oct 30 13:03:10.902873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902807    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-76hdh\""
Oct 30 13:03:10.902873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902812    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 13:03:10.902873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902768    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.902873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902853    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.902873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902768    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zbfrs\""
Oct 30 13:03:10.902873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902870    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902888    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-r278v\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902901    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902902    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902909    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-prxcs\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902921    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902934    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vz5h4\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902980    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.902992    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.903021    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.903035    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.903043    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.903053    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 13:03:10.903129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.903060    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 13:03:10.904519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.904506    4291 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.904683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.904668    4291 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.924452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.924437    4291 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:10.924503 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:10.924473    4291 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-36.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.927630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.927610    4291 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 13:03:10.927727 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:10.927677    4291 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-36.ec2.internal"
Oct 30 13:03:10.946260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:10.946229    4291 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 13:03:11.000132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000098    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-run\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.000132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000133    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-var-lib-kubelet\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.000362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000160    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20cc5462-99f5-410c-97d6-b414507f297a-hosts-file\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:03:11.000362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000211    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-kubelet\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.000362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000252    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cni-binary-copy\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.000362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000277    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.000362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000300    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-lib-modules\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.000362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000319    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5zlw\" (UniqueName: \"kubernetes.io/projected/c56d84ee-39fa-48a9-9426-60b730cfaccc-kube-api-access-k5zlw\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.000362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000354    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.000682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000378    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-conf-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.000682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000536    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-system-cni-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.000682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000671    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cni-binary-copy\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.000807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000681    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/37b49c3d-d533-4d1f-898b-2c2fa83e2126-original-pull-secret\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.000807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000671    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.000986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.000969    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-cnibin\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.001024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001010    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-tuned\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.001054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001038    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ec2afa3-085d-4e46-8488-60154d80ec91-serviceca\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:03:11.001088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001064    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-socket-dir-parent\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.001119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001088    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-socket-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.001119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001095    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-tuned\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.001119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001112    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-registration-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.001224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001136    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-sys-fs\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.001224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001164    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-sys\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.001224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001183    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-config\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001201    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-multus-certs\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001227    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001244    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-systemd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001270    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-netd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001289    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-tmp\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001326    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-var-lib-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001370    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-config\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001376    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-ovn\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001405    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001424    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001437    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c56d84ee-39fa-48a9-9426-60b730cfaccc-tmp\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001433    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001457    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cnibin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001505    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001549    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4gz6\" (UniqueName: \"kubernetes.io/projected/b331a27e-e822-4419-b000-bf767335246a-kube-api-access-k4gz6\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001608    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-modprobe-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001660    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-systemd-units\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.001712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001707    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-agent-certs\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001725    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/7ec2afa3-085d-4e46-8488-60154d80ec91-serviceca\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001740    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-k8s-cni-cncf-io\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001750    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/02f4d975-e0f4-456c-a85e-2a1389468f5c-cni-binary-copy\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001776    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-bin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001830    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnjj8\" (UniqueName: \"kubernetes.io/projected/9bc491f0-f83f-48fa-8f83-92f88f9f9467-kube-api-access-jnjj8\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001846    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001869    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-kubernetes\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001888    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-systemd\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001921    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8ww8\" (UniqueName: \"kubernetes.io/projected/ebb46e24-3980-4909-83fb-bf22d54b5031-kube-api-access-t8ww8\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.001967    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-agent-certs\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002010    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-log-socket\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002028    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovn-node-metrics-cert\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002071    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-system-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002087    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-conf\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002134    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-host\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002174    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cc5462-99f5-410c-97d6-b414507f297a-tmp-dir\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002205    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-node-log\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002250    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/20cc5462-99f5-410c-97d6-b414507f297a-tmp-dir\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002279    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-script-lib\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002370    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovn-node-metrics-cert\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002401    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-etc-kubernetes\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002424    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002427    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-ovnkube-script-lib\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002446    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:11.002489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002461    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-slash\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002501    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002541    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-device-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002576    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ftjq\" (UniqueName: \"kubernetes.io/projected/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kube-api-access-6ftjq\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002614    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b331a27e-e822-4419-b000-bf767335246a-iptables-alerter-script\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002659    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b331a27e-e822-4419-b000-bf767335246a-host-slash\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002755    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002778    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec2afa3-085d-4e46-8488-60154d80ec91-host\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002814    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b331a27e-e822-4419-b000-bf767335246a-iptables-alerter-script\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002819    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9pbg\" (UniqueName: \"kubernetes.io/projected/7ec2afa3-085d-4e46-8488-60154d80ec91-kube-api-access-h9pbg\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002846    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-kubelet-config\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002860    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-dbus\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002905    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-netns\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002975    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysconfig\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.002999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.002994    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x75s\" (UniqueName: \"kubernetes.io/projected/20cc5462-99f5-410c-97d6-b414507f297a-kube-api-access-6x75s\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003023    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-etc-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003046    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-konnectivity-ca\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003063    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-kubelet\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003082    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003096    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-netns\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003111    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003153    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-hostroot\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003181    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-daemon-config\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003265    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3-konnectivity-ca\") pod \"konnectivity-agent-xnlf9\" (UID: \"0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3\") " pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003310    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-etc-selinux\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003383    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-bin\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003411    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-multus\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003421    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-daemon-config\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003449    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzhw\" (UniqueName: \"kubernetes.io/projected/02f4d975-e0f4-456c-a85e-2a1389468f5c-kube-api-access-qjzhw\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003483    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-env-overrides\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003568    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6l8z4\" (UniqueName: \"kubernetes.io/projected/723b2da3-c191-420c-b6d1-7f8a4a9806cd-kube-api-access-6l8z4\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.003577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.003578    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/723b2da3-c191-420c-b6d1-7f8a4a9806cd-env-overrides\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.017817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.017801    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5zlw\" (UniqueName: \"kubernetes.io/projected/c56d84ee-39fa-48a9-9426-60b730cfaccc-kube-api-access-k5zlw\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.018692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.018668    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8ww8\" (UniqueName: \"kubernetes.io/projected/ebb46e24-3980-4909-83fb-bf22d54b5031-kube-api-access-t8ww8\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:11.018783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.018698    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4gz6\" (UniqueName: \"kubernetes.io/projected/b331a27e-e822-4419-b000-bf767335246a-kube-api-access-k4gz6\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:03:11.018924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.018909    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnjj8\" (UniqueName: \"kubernetes.io/projected/9bc491f0-f83f-48fa-8f83-92f88f9f9467-kube-api-access-jnjj8\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.019775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.019749    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ftjq\" (UniqueName: \"kubernetes.io/projected/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kube-api-access-6ftjq\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.019862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.019839    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x75s\" (UniqueName: \"kubernetes.io/projected/20cc5462-99f5-410c-97d6-b414507f297a-kube-api-access-6x75s\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:03:11.019926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.019858    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9pbg\" (UniqueName: \"kubernetes.io/projected/7ec2afa3-085d-4e46-8488-60154d80ec91-kube-api-access-h9pbg\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:03:11.020358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.020337    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnz2d\" (UniqueName: \"kubernetes.io/projected/28913fb0-2984-4375-8fe0-e15682bde559-kube-api-access-gnz2d\") pod \"network-check-target-8542k\" (UID: \"28913fb0-2984-4375-8fe0-e15682bde559\") " pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:11.020858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.020839    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6l8z4\" (UniqueName: \"kubernetes.io/projected/723b2da3-c191-420c-b6d1-7f8a4a9806cd-kube-api-access-6l8z4\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.020904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.020841    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzhw\" (UniqueName: \"kubernetes.io/projected/02f4d975-e0f4-456c-a85e-2a1389468f5c-kube-api-access-qjzhw\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.104392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104307    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-etc-kubernetes\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.104392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104341    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.104392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104364    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:11.104392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104380    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-slash\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104396    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104420    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-device-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104421    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-etc-kubernetes\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104445    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b331a27e-e822-4419-b000-bf767335246a-host-slash\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104456    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-slash\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104465    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-tuning-conf-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104473    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104472    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104502    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-device-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104511    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b331a27e-e822-4419-b000-bf767335246a-host-slash\") pod \"iptables-alerter-nlmnl\" (UID: \"b331a27e-e822-4419-b000-bf767335246a\") " pod="openshift-network-operator/iptables-alerter-nlmnl"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104525    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec2afa3-085d-4e46-8488-60154d80ec91-host\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104548    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-kubelet-config\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104564    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-dbus\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104584    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-netns\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104589    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104586    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7ec2afa3-085d-4e46-8488-60154d80ec91-host\") pod \"node-ca-gr5nl\" (UID: \"7ec2afa3-085d-4e46-8488-60154d80ec91\") " pod="openshift-image-registry/node-ca-gr5nl"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104607    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysconfig\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104628    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-netns\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104650    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-etc-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104656    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-kubelet-config\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104650    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/37b49c3d-d533-4d1f-898b-2c2fa83e2126-dbus\") pod \"global-pull-secret-syncer-j4n8s\" (UID: \"37b49c3d-d533-4d1f-898b-2c2fa83e2126\") " pod="kube-system/global-pull-secret-syncer-j4n8s"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104679    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysconfig\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104679    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-kubelet\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104711    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-kubelet\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.104729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104734    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104687    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-etc-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104757    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-netns\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104779    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104791    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-netns\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104800    4291 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104837    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-hostroot\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104805    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-hostroot\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104901    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-etc-selinux\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104917    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-bin\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104931    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-multus\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104948    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-run\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104964    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-var-lib-kubelet\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104982    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-bin\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104988    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-etc-selinux\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.104990    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20cc5462-99f5-410c-97d6-b414507f297a-hosts-file\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105002    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-multus\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105023    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-kubelet\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105030    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-run\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105033    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/20cc5462-99f5-410c-97d6-b414507f297a-hosts-file\") pod \"node-resolver-5sstj\" (UID: \"20cc5462-99f5-410c-97d6-b414507f297a\") " pod="openshift-dns/node-resolver-5sstj"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105040    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-var-lib-kubelet\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105056    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-lib-modules\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105066    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-kubelet\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105073    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-conf-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105096    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-system-cni-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105114    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-cnibin\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105142    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-lib-modules\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105138    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-conf-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105140    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-socket-dir-parent\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105166    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-system-cni-dir\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105192    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-socket-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105218    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-registration-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105225    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-socket-dir-parent\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105194    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/02f4d975-e0f4-456c-a85e-2a1389468f5c-cnibin\") pod \"multus-additional-cni-plugins-c2nts\" (UID: \"02f4d975-e0f4-456c-a85e-2a1389468f5c\") " pod="openshift-multus/multus-additional-cni-plugins-c2nts"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105242    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-sys-fs\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105265    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-sys\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105270    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-registration-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105289    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-multus-certs\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105294    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-sys-fs\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105268    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-socket-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105310    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-sys\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105314    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-systemd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105339    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-netd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105349    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-systemd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105339    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-multus-certs\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105362    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-var-lib-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105376    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-cni-netd\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105384    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-ovn\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105405    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-var-lib-openvswitch\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105409    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105407    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-run-ovn\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105434    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105434    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-host-run-ovn-kubernetes\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105458    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cnibin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105476    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-multus-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105482    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-modprobe-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105503    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-cnibin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105508    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-systemd-units\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105536    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-k8s-cni-cncf-io\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105542    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-systemd-units\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105560    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-bin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105581    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-modprobe-d\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105584    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105608    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-kubernetes\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105612    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-run-k8s-cni-cncf-io\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105626    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-host-var-lib-cni-bin\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.105604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105652    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-systemd\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105675    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/da470b83-8d33-4fe6-a4c9-2c3a0b85d060-kubelet-dir\") pod \"aws-ebs-csi-driver-node-64rvg\" (UID: \"da470b83-8d33-4fe6-a4c9-2c3a0b85d060\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-64rvg"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105678    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-kubernetes\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105679    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-log-socket\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105716    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-systemd\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105721    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-log-socket\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105728    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-system-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105747    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-conf\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105760    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-host\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105775    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-node-log\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105823    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-host\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105822    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/723b2da3-c191-420c-b6d1-7f8a4a9806cd-node-log\") pod \"ovnkube-node-28lqx\" (UID: \"723b2da3-c191-420c-b6d1-7f8a4a9806cd\") " pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105849    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9bc491f0-f83f-48fa-8f83-92f88f9f9467-system-cni-dir\") pod \"multus-lz7gh\" (UID: \"9bc491f0-f83f-48fa-8f83-92f88f9f9467\") " pod="openshift-multus/multus-lz7gh"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.105867    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c56d84ee-39fa-48a9-9426-60b730cfaccc-etc-sysctl-conf\") pod \"tuned-k52kk\" (UID: \"c56d84ee-39fa-48a9-9426-60b730cfaccc\") " pod="openshift-cluster-node-tuning-operator/tuned-k52kk"
Oct 30 13:03:11.107476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.106977    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ebb46e24-3980-4909-83fb-bf22d54b5031-metrics-certs\") pod \"network-metrics-daemon-66rk6\" (UID: \"ebb46e24-3980-4909-83fb-bf22d54b5031\") " pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:11.175484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.175465    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66rk6"
Oct 30 13:03:11.490080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.489998    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-66rk6"]
Oct 30 13:03:11.493457 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:11.493430    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb46e24_3980_4909_83fb_bf22d54b5031.slice/crio-10cbbdab75e1fc21ad26f6fba5b154ec2ad7aa68caa4fb29d447b68aac0649dc WatchSource:0}: Error finding container 10cbbdab75e1fc21ad26f6fba5b154ec2ad7aa68caa4fb29d447b68aac0649dc: Status 404 returned error can't find the container with id 10cbbdab75e1fc21ad26f6fba5b154ec2ad7aa68caa4fb29d447b68aac0649dc
Oct 30 13:03:11.495623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.495604    4291 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:03:11.909006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:11.908872    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66rk6" event={"ID":"ebb46e24-3980-4909-83fb-bf22d54b5031","Type":"ContainerStarted","Data":"10cbbdab75e1fc21ad26f6fba5b154ec2ad7aa68caa4fb29d447b68aac0649dc"}
Oct 30 13:03:12.913254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:12.913163    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66rk6" event={"ID":"ebb46e24-3980-4909-83fb-bf22d54b5031","Type":"ContainerStarted","Data":"bd5b4096fbb4d36cbfb9160411fc7c1116d7f30f98ee40e9490c66eb5cd8133d"}
Oct 30 13:03:12.913254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:12.913223    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66rk6" event={"ID":"ebb46e24-3980-4909-83fb-bf22d54b5031","Type":"ContainerStarted","Data":"a33653dc56b3d1c1dd991772c8f46af58dffb9e8dc79f94f02a9805bbc2d5fad"}
Oct 30 13:03:15.594881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:15.594849    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xnlf9"
Oct 30 13:03:16.761973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.761941    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"]
Oct 30 13:03:16.763842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.763825    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-rmscn"]
Oct 30 13:03:16.763980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.763961    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"
Oct 30 13:03:16.765838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.765820    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:16.781036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.781015    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-wdxst"]
Oct 30 13:03:16.782510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.782479    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:16.786503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.786479    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Oct 30 13:03:16.786694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.786677    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-bs47v\""
Oct 30 13:03:16.787218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.787158    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"]
Oct 30 13:03:16.795707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.795689    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rmscn"]
Oct 30 13:03:16.798538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.798518    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 13:03:16.798538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.798527    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-6t7p9\""
Oct 30 13:03:16.798678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.798541    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 13:03:16.805599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.805580    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 13:03:16.805599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.805589    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 13:03:16.818429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.818405    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wdxst"]
Oct 30 13:03:16.829166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.829137    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m8257"]
Oct 30 13:03:16.831104 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.831087    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:16.833334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.833307    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c18f5676-dc4b-45af-a3af-af8db22da5b2-tmp\") pod \"downloads-6dd789c8b7-rmscn\" (UID: \"c18f5676-dc4b-45af-a3af-af8db22da5b2\") " pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:16.833418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.833343    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2scm\" (UniqueName: \"kubernetes.io/projected/c18f5676-dc4b-45af-a3af-af8db22da5b2-kube-api-access-k2scm\") pod \"downloads-6dd789c8b7-rmscn\" (UID: \"c18f5676-dc4b-45af-a3af-af8db22da5b2\") " pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:16.833527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.833499    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9becd096-71d7-431d-a6f5-efe50f052a16-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mmlnt\" (UID: \"9becd096-71d7-431d-a6f5-efe50f052a16\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"
Oct 30 13:03:16.856325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.856288    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m8257"]
Oct 30 13:03:16.856747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.856726    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-nsrbd"]
Oct 30 13:03:16.859048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.859028    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7964b8d7bd-cpfkm"]
Oct 30 13:03:16.859206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.859188    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:16.860303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.860283    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 13:03:16.860382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.860283    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 13:03:16.860382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.860285    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 13:03:16.861090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.861072    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.877182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.877163    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 13:03:16.882108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.882074    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nsrbd"]
Oct 30 13:03:16.884899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.884878    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7964b8d7bd-cpfkm"]
Oct 30 13:03:16.900102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900065    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 13:03:16.900261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900232    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fx9p2\""
Oct 30 13:03:16.900320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900241    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Oct 30 13:03:16.900357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900348    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 13:03:16.900401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900371    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Oct 30 13:03:16.900401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900387    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Oct 30 13:03:16.900474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900408    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-tdfvc\""
Oct 30 13:03:16.900474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900422    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9mrzf\""
Oct 30 13:03:16.900474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900445    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 13:03:16.900588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900504    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-hwzrd\""
Oct 30 13:03:16.900588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.900525    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Oct 30 13:03:16.934838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.934800    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9becd096-71d7-431d-a6f5-efe50f052a16-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mmlnt\" (UID: \"9becd096-71d7-431d-a6f5-efe50f052a16\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"
Oct 30 13:03:16.934838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.934849    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1517a6-e078-4162-88ca-27a8da3984d1-registry-certificates\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.935054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.934874    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d72e0d15-ac35-467a-a98c-057ac0eedc0b-crio-socket\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:16.935054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.934916    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72e0d15-ac35-467a-a98c-057ac0eedc0b-data-volume\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:16.935054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.934933    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ecd77b-60e5-45f0-b79a-6f9d429f0820-metrics-tls\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:16.935054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935029    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98f18755-7c4a-4129-9f2c-78720e88abb2-cert\") pod \"ingress-canary-wdxst\" (UID: \"98f18755-7c4a-4129-9f2c-78720e88abb2\") " pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:16.935187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935051    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1517a6-e078-4162-88ca-27a8da3984d1-image-registry-private-configuration\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.935187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935076    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c18f5676-dc4b-45af-a3af-af8db22da5b2-tmp\") pod \"downloads-6dd789c8b7-rmscn\" (UID: \"c18f5676-dc4b-45af-a3af-af8db22da5b2\") " pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:16.935187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nk8jt\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-kube-api-access-nk8jt\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.935187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935146    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ecd77b-60e5-45f0-b79a-6f9d429f0820-config-volume\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:16.935187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935162    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ecd77b-60e5-45f0-b79a-6f9d429f0820-tmp-dir\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:16.935187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935179    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2scm\" (UniqueName: \"kubernetes.io/projected/c18f5676-dc4b-45af-a3af-af8db22da5b2-kube-api-access-k2scm\") pod \"downloads-6dd789c8b7-rmscn\" (UID: \"c18f5676-dc4b-45af-a3af-af8db22da5b2\") " pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:16.935449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935203    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1517a6-e078-4162-88ca-27a8da3984d1-ca-trust-extracted\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.935449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935280    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-registry-tls\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.935449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935326    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1517a6-e078-4162-88ca-27a8da3984d1-trusted-ca\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.935449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935392    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d72e0d15-ac35-467a-a98c-057ac0eedc0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:16.935676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935461    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9trnq\" (UniqueName: \"kubernetes.io/projected/d72e0d15-ac35-467a-a98c-057ac0eedc0b-kube-api-access-9trnq\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:16.935676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935511    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-plxk9\" (UniqueName: \"kubernetes.io/projected/95ecd77b-60e5-45f0-b79a-6f9d429f0820-kube-api-access-plxk9\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:16.935676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935551    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rj66v\" (UniqueName: \"kubernetes.io/projected/98f18755-7c4a-4129-9f2c-78720e88abb2-kube-api-access-rj66v\") pod \"ingress-canary-wdxst\" (UID: \"98f18755-7c4a-4129-9f2c-78720e88abb2\") " pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:16.935676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935580    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d72e0d15-ac35-467a-a98c-057ac0eedc0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:16.935676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935608    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1517a6-e078-4162-88ca-27a8da3984d1-installation-pull-secrets\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.935676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935621    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c18f5676-dc4b-45af-a3af-af8db22da5b2-tmp\") pod \"downloads-6dd789c8b7-rmscn\" (UID: \"c18f5676-dc4b-45af-a3af-af8db22da5b2\") " pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:16.935676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.935658    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-bound-sa-token\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:16.937395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.937371    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/9becd096-71d7-431d-a6f5-efe50f052a16-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-mmlnt\" (UID: \"9becd096-71d7-431d-a6f5-efe50f052a16\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"
Oct 30 13:03:16.962495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:16.962470    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2scm\" (UniqueName: \"kubernetes.io/projected/c18f5676-dc4b-45af-a3af-af8db22da5b2-kube-api-access-k2scm\") pod \"downloads-6dd789c8b7-rmscn\" (UID: \"c18f5676-dc4b-45af-a3af-af8db22da5b2\") " pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:17.036577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036480    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nk8jt\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-kube-api-access-nk8jt\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.036577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036531    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ecd77b-60e5-45f0-b79a-6f9d429f0820-config-volume\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.036577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036551    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ecd77b-60e5-45f0-b79a-6f9d429f0820-tmp-dir\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.036577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036571    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1517a6-e078-4162-88ca-27a8da3984d1-ca-trust-extracted\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036597    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-registry-tls\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036618    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1517a6-e078-4162-88ca-27a8da3984d1-trusted-ca\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036757    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d72e0d15-ac35-467a-a98c-057ac0eedc0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036808    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9trnq\" (UniqueName: \"kubernetes.io/projected/d72e0d15-ac35-467a-a98c-057ac0eedc0b-kube-api-access-9trnq\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036836    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-plxk9\" (UniqueName: \"kubernetes.io/projected/95ecd77b-60e5-45f0-b79a-6f9d429f0820-kube-api-access-plxk9\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036866    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rj66v\" (UniqueName: \"kubernetes.io/projected/98f18755-7c4a-4129-9f2c-78720e88abb2-kube-api-access-rj66v\") pod \"ingress-canary-wdxst\" (UID: \"98f18755-7c4a-4129-9f2c-78720e88abb2\") " pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036894    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d72e0d15-ac35-467a-a98c-057ac0eedc0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.036931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036920    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1517a6-e078-4162-88ca-27a8da3984d1-installation-pull-secrets\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036943    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-bound-sa-token\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036992    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1517a6-e078-4162-88ca-27a8da3984d1-registry-certificates\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.036998    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/95ecd77b-60e5-45f0-b79a-6f9d429f0820-tmp-dir\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037017    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/eb1517a6-e078-4162-88ca-27a8da3984d1-ca-trust-extracted\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037033    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d72e0d15-ac35-467a-a98c-057ac0eedc0b-crio-socket\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037058    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72e0d15-ac35-467a-a98c-057ac0eedc0b-data-volume\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037085    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ecd77b-60e5-45f0-b79a-6f9d429f0820-metrics-tls\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037121    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98f18755-7c4a-4129-9f2c-78720e88abb2-cert\") pod \"ingress-canary-wdxst\" (UID: \"98f18755-7c4a-4129-9f2c-78720e88abb2\") " pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:17.037313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037155    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1517a6-e078-4162-88ca-27a8da3984d1-image-registry-private-configuration\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.037781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037320    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d72e0d15-ac35-467a-a98c-057ac0eedc0b-crio-socket\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.037781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037498    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d72e0d15-ac35-467a-a98c-057ac0eedc0b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.037916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.037887    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/eb1517a6-e078-4162-88ca-27a8da3984d1-trusted-ca\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.038474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.038155    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/95ecd77b-60e5-45f0-b79a-6f9d429f0820-config-volume\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.038474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.038234    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/eb1517a6-e078-4162-88ca-27a8da3984d1-registry-certificates\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.038474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.038414    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72e0d15-ac35-467a-a98c-057ac0eedc0b-data-volume\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.039839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.039804    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-registry-tls\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.039938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.039846    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/eb1517a6-e078-4162-88ca-27a8da3984d1-image-registry-private-configuration\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.039938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.039858    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d72e0d15-ac35-467a-a98c-057ac0eedc0b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.040127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.040110    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/95ecd77b-60e5-45f0-b79a-6f9d429f0820-metrics-tls\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.040312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.040290    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/98f18755-7c4a-4129-9f2c-78720e88abb2-cert\") pod \"ingress-canary-wdxst\" (UID: \"98f18755-7c4a-4129-9f2c-78720e88abb2\") " pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:17.040446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.040430    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/eb1517a6-e078-4162-88ca-27a8da3984d1-installation-pull-secrets\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.056491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.056457    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nk8jt\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-kube-api-access-nk8jt\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.056624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.056459    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rj66v\" (UniqueName: \"kubernetes.io/projected/98f18755-7c4a-4129-9f2c-78720e88abb2-kube-api-access-rj66v\") pod \"ingress-canary-wdxst\" (UID: \"98f18755-7c4a-4129-9f2c-78720e88abb2\") " pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:17.056624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.056516    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-plxk9\" (UniqueName: \"kubernetes.io/projected/95ecd77b-60e5-45f0-b79a-6f9d429f0820-kube-api-access-plxk9\") pod \"dns-default-nsrbd\" (UID: \"95ecd77b-60e5-45f0-b79a-6f9d429f0820\") " pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.056624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.056562    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9trnq\" (UniqueName: \"kubernetes.io/projected/d72e0d15-ac35-467a-a98c-057ac0eedc0b-kube-api-access-9trnq\") pod \"insights-runtime-extractor-m8257\" (UID: \"d72e0d15-ac35-467a-a98c-057ac0eedc0b\") " pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.056893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.056872    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/eb1517a6-e078-4162-88ca-27a8da3984d1-bound-sa-token\") pod \"image-registry-7964b8d7bd-cpfkm\" (UID: \"eb1517a6-e078-4162-88ca-27a8da3984d1\") " pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.075541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.075511    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"
Oct 30 13:03:17.080063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.080046    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:17.094982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.094956    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-wdxst"
Oct 30 13:03:17.144239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.143355    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m8257"
Oct 30 13:03:17.175607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.174519    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:17.177383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.177311    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:17.214295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.214241    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:17.376085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.376055    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"]
Oct 30 13:03:17.378918 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:17.378895    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9becd096_71d7_431d_a6f5_efe50f052a16.slice/crio-729ea41247ce8005e8b52d9fb8eb5ffe7662c50e0245359b6879e3b3d5f90b46 WatchSource:0}: Error finding container 729ea41247ce8005e8b52d9fb8eb5ffe7662c50e0245359b6879e3b3d5f90b46: Status 404 returned error can't find the container with id 729ea41247ce8005e8b52d9fb8eb5ffe7662c50e0245359b6879e3b3d5f90b46
Oct 30 13:03:17.392741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.392511    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-rmscn"]
Oct 30 13:03:17.396041 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:17.396014    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18f5676_dc4b_45af_a3af_af8db22da5b2.slice/crio-fa90fde4be4a612c75349569c1ccf0836a56820c30fe00ccd009e76a87ced6b4 WatchSource:0}: Error finding container fa90fde4be4a612c75349569c1ccf0836a56820c30fe00ccd009e76a87ced6b4: Status 404 returned error can't find the container with id fa90fde4be4a612c75349569c1ccf0836a56820c30fe00ccd009e76a87ced6b4
Oct 30 13:03:17.418332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.418274    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-wdxst"]
Oct 30 13:03:17.422306 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:17.422278    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f18755_7c4a_4129_9f2c_78720e88abb2.slice/crio-a6075aa30c7df334b68f149e4562a40ccb2dbaf3940d731e1bc571bc9af38a49 WatchSource:0}: Error finding container a6075aa30c7df334b68f149e4562a40ccb2dbaf3940d731e1bc571bc9af38a49: Status 404 returned error can't find the container with id a6075aa30c7df334b68f149e4562a40ccb2dbaf3940d731e1bc571bc9af38a49
Oct 30 13:03:17.489276 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.489253    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m8257"]
Oct 30 13:03:17.492957 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:17.492929    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72e0d15_ac35_467a_a98c_057ac0eedc0b.slice/crio-e1158dcbbbe0a4b608ca385845b8a996673e3e3bff0632a1f7e3b686fd273fc7 WatchSource:0}: Error finding container e1158dcbbbe0a4b608ca385845b8a996673e3e3bff0632a1f7e3b686fd273fc7: Status 404 returned error can't find the container with id e1158dcbbbe0a4b608ca385845b8a996673e3e3bff0632a1f7e3b686fd273fc7
Oct 30 13:03:17.514672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.514648    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-nsrbd"]
Oct 30 13:03:17.518334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.518313    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7964b8d7bd-cpfkm"]
Oct 30 13:03:17.522915 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:17.522893    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ecd77b_60e5_45f0_b79a_6f9d429f0820.slice/crio-503bd54f34c88221f2255efb74b98a4bb30d3c12c4877a4f6e307a2d0aa17f67 WatchSource:0}: Error finding container 503bd54f34c88221f2255efb74b98a4bb30d3c12c4877a4f6e307a2d0aa17f67: Status 404 returned error can't find the container with id 503bd54f34c88221f2255efb74b98a4bb30d3c12c4877a4f6e307a2d0aa17f67
Oct 30 13:03:17.523088 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:17.523067    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb1517a6_e078_4162_88ca_27a8da3984d1.slice/crio-c95446b9570f5c984fc88da01a79ffc64a0dac1db1b52f21089f45b568818756 WatchSource:0}: Error finding container c95446b9570f5c984fc88da01a79ffc64a0dac1db1b52f21089f45b568818756: Status 404 returned error can't find the container with id c95446b9570f5c984fc88da01a79ffc64a0dac1db1b52f21089f45b568818756
Oct 30 13:03:17.934181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.934143    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt" event={"ID":"9becd096-71d7-431d-a6f5-efe50f052a16","Type":"ContainerStarted","Data":"729ea41247ce8005e8b52d9fb8eb5ffe7662c50e0245359b6879e3b3d5f90b46"}
Oct 30 13:03:17.936696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.936665    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsrbd" event={"ID":"95ecd77b-60e5-45f0-b79a-6f9d429f0820","Type":"ContainerStarted","Data":"503bd54f34c88221f2255efb74b98a4bb30d3c12c4877a4f6e307a2d0aa17f67"}
Oct 30 13:03:17.938565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.938540    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8257" event={"ID":"d72e0d15-ac35-467a-a98c-057ac0eedc0b","Type":"ContainerStarted","Data":"fd791c8e8eacfcc86c30fc219fdfd2565ed22a395b6b094c0ba202d0b30b90af"}
Oct 30 13:03:17.938698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.938605    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8257" event={"ID":"d72e0d15-ac35-467a-a98c-057ac0eedc0b","Type":"ContainerStarted","Data":"e1158dcbbbe0a4b608ca385845b8a996673e3e3bff0632a1f7e3b686fd273fc7"}
Oct 30 13:03:17.940286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.940250    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rmscn" event={"ID":"c18f5676-dc4b-45af-a3af-af8db22da5b2","Type":"ContainerStarted","Data":"fa90fde4be4a612c75349569c1ccf0836a56820c30fe00ccd009e76a87ced6b4"}
Oct 30 13:03:17.942369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.942342    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm" event={"ID":"eb1517a6-e078-4162-88ca-27a8da3984d1","Type":"ContainerStarted","Data":"c40a3af465ae8aa17c8ce90ed298ab79736a337b1e103c8f6d89d0d2ee791e6f"}
Oct 30 13:03:17.942485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.942389    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm" event={"ID":"eb1517a6-e078-4162-88ca-27a8da3984d1","Type":"ContainerStarted","Data":"c95446b9570f5c984fc88da01a79ffc64a0dac1db1b52f21089f45b568818756"}
Oct 30 13:03:17.945131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:17.945082    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wdxst" event={"ID":"98f18755-7c4a-4129-9f2c-78720e88abb2","Type":"ContainerStarted","Data":"a6075aa30c7df334b68f149e4562a40ccb2dbaf3940d731e1bc571bc9af38a49"}
Oct 30 13:03:18.271066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.270176    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:18.308548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.308497    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm" podStartSLOduration=2.308480353 podStartE2EDuration="2.308480353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:17.983941627 +0000 UTC m=+8.244179862" watchObservedRunningTime="2026-10-30 13:03:18.308480353 +0000 UTC m=+8.568718582"
Oct 30 13:03:18.953321 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.953026    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8257" event={"ID":"d72e0d15-ac35-467a-a98c-057ac0eedc0b","Type":"ContainerStarted","Data":"93ef12f128970cc86c2f4ef1721c51bf9d5c5a745a95510fafbc7621c59d9ea4"}
Oct 30 13:03:18.956937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.955993    4291 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:03:18.956937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.956716    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt" event={"ID":"9becd096-71d7-431d-a6f5-efe50f052a16","Type":"ContainerStarted","Data":"fbb2e80a5bbc91e6521e4ca509a7c8c051845af925743969165be1070b5b3b6c"}
Oct 30 13:03:18.963667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.963202    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:18.963667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.963397    4291 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:03:18.996562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:18.996501    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt" podStartSLOduration=1.626684617 podStartE2EDuration="2.996484337s" totalImagesPullingTime="1.36979972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:18.995820014 +0000 UTC m=+9.256058250" watchObservedRunningTime="2026-10-30 13:03:18.996484337 +0000 UTC m=+9.256722574"
Oct 30 13:03:19.961750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:19.961718    4291 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 13:03:20.048475 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:20.048439    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:20.541317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:20.541230    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"
Oct 30 13:03:20.547669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:20.547623    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-mmlnt"
Oct 30 13:03:20.966806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:20.966778    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsrbd" event={"ID":"95ecd77b-60e5-45f0-b79a-6f9d429f0820","Type":"ContainerStarted","Data":"7236a589dc814907479ab404e90dc185140ab3dadbbf21ec553797e33cf6bd12"}
Oct 30 13:03:20.967149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:20.966845    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-nsrbd" event={"ID":"95ecd77b-60e5-45f0-b79a-6f9d429f0820","Type":"ContainerStarted","Data":"43db4190e1da325a3460a3d2995107e109e6f1b1ccb74fa835285176237bea7c"}
Oct 30 13:03:20.967149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:20.966909    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:20.970024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:20.969957    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m8257" event={"ID":"d72e0d15-ac35-467a-a98c-057ac0eedc0b","Type":"ContainerStarted","Data":"6b40d412e9b776fa0f67fb6b87b50efd27961d52a0900174b68ec44803a3420e"}
Oct 30 13:03:21.007808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:21.007760    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-nsrbd" podStartSLOduration=2.537301167 podStartE2EDuration="5.007747226s" totalImagesPullingTime="2.470446059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:21.007272124 +0000 UTC m=+11.267510374" watchObservedRunningTime="2026-10-30 13:03:21.007747226 +0000 UTC m=+11.267985470"
Oct 30 13:03:21.975351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:21.974889    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-wdxst" event={"ID":"98f18755-7c4a-4129-9f2c-78720e88abb2","Type":"ContainerStarted","Data":"0606cf69a01133bb017451a935213b7e70966409a2e01aeb21a67cc5eb01d021"}
Oct 30 13:03:22.013854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:22.013811    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-wdxst" podStartSLOduration=2.516937162 podStartE2EDuration="6.013798762s" totalImagesPullingTime="3.4968616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:22.013662853 +0000 UTC m=+12.273901089" watchObservedRunningTime="2026-10-30 13:03:22.013798762 +0000 UTC m=+12.274036996"
Oct 30 13:03:22.014146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:22.014112    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m8257" podStartSLOduration=3.482930863 podStartE2EDuration="6.014101873s" totalImagesPullingTime="2.53117101s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 13:03:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:21.049350301 +0000 UTC m=+11.309588539" watchObservedRunningTime="2026-10-30 13:03:22.014101873 +0000 UTC m=+12.274340107"
Oct 30 13:03:26.680078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.680041    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-pdh7p"]
Oct 30 13:03:26.683687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.683656    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.708716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.708690    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-28lqx"
Oct 30 13:03:26.722344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.722315    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 13:03:26.722344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.722315    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 13:03:26.722544 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.722414    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 13:03:26.722544 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.722492    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 13:03:26.775066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.775038    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 13:03:26.775259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.775141    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 13:03:26.775439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.775422    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-pbv95\""
Oct 30 13:03:26.816109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816073    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-textfile\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816119    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/6df08611-a79d-4a59-9371-29dfa510b9be-kube-api-access-gtzdk\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816289    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816329    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-tls\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816454    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-accelerators-collector-config\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816501    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-sys\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816543    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-root\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816571    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df08611-a79d-4a59-9371-29dfa510b9be-metrics-client-ca\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.816667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.816608    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-wtmp\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.917879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.917848    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/6df08611-a79d-4a59-9371-29dfa510b9be-kube-api-access-gtzdk\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.917898    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.917921    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-tls\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.917954    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-accelerators-collector-config\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.917981    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-sys\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918008    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-root\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918033    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df08611-a79d-4a59-9371-29dfa510b9be-metrics-client-ca\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918055    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-wtmp\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918095    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-textfile\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918223    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-sys\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918452    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-textfile\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.918555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918514    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-root\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.919207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.918683    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-wtmp\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.919207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.919162    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6df08611-a79d-4a59-9371-29dfa510b9be-metrics-client-ca\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.919207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.919170    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-accelerators-collector-config\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.920863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.920831    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-tls\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.924426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.924402    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/6df08611-a79d-4a59-9371-29dfa510b9be-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.941974 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.941924    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtzdk\" (UniqueName: \"kubernetes.io/projected/6df08611-a79d-4a59-9371-29dfa510b9be-kube-api-access-gtzdk\") pod \"node-exporter-pdh7p\" (UID: \"6df08611-a79d-4a59-9371-29dfa510b9be\") " pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:26.998127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:26.998092    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-pdh7p"
Oct 30 13:03:27.009271 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:27.009234    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6df08611_a79d_4a59_9371_29dfa510b9be.slice/crio-c1d568c85c4b88946326fa8779c2f473960f1399f239125b4ec2de7e3c1fdd26 WatchSource:0}: Error finding container c1d568c85c4b88946326fa8779c2f473960f1399f239125b4ec2de7e3c1fdd26: Status 404 returned error can't find the container with id c1d568c85c4b88946326fa8779c2f473960f1399f239125b4ec2de7e3c1fdd26
Oct 30 13:03:27.998218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:27.998182    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdh7p" event={"ID":"6df08611-a79d-4a59-9371-29dfa510b9be","Type":"ContainerStarted","Data":"101d37bac86d206496d198bb53a69caae0a1edb321ea6acbccee3ead6e2391ae"}
Oct 30 13:03:27.998668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:27.998254    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdh7p" event={"ID":"6df08611-a79d-4a59-9371-29dfa510b9be","Type":"ContainerStarted","Data":"c1d568c85c4b88946326fa8779c2f473960f1399f239125b4ec2de7e3c1fdd26"}
Oct 30 13:03:28.109245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.109211    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 13:03:28.112117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.112098    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.126772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.126711    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 13:03:28.126772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.126727    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 13:03:28.126772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.126718    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 13:03:28.126772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.126727    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 13:03:28.127035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.126785    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 13:03:28.127035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.126824    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 13:03:28.127035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.126825    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 13:03:28.138753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.138663    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 13:03:28.151048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.151022    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 13:03:28.226670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226620    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.226866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226685    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.226866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226717    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.226866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226743    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.226866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226746    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-9ttmx\""
Oct 30 13:03:28.226866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226765    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 13:03:28.226866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226767    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226939    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.226986    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsx4x\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-kube-api-access-zsx4x\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.227013    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-out\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.227101    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-web-config\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.227124    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.227150    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.227187    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.227453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.227220    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.327937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.327895    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsx4x\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-kube-api-access-zsx4x\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.327945    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-out\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.327978    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-web-config\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328000    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328031    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328056    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328079    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328113    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328138    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328160    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328182    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328205    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328238    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.328721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.328679    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.330011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.329078    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.330011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.329962    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.331624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.331592    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-volume\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.332138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.332052    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.332267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.332248    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-web-config\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.332346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.332311    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.332587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.332564    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-out\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.332925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.332901    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.333143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.333100    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.333326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.333301    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.333544 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.333528    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.347535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.347509    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsx4x\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-kube-api-access-zsx4x\") pod \"alertmanager-main-0\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.432575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.432496    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:03:28.776234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.776165    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 13:03:28.779629 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:28.779602    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a87588_274a_4c9c_89a6_3fa864fc18ec.slice/crio-6825ed2e5df7fa22149e8d1111b464bd1ca6978608956be93c45db58a31dff14 WatchSource:0}: Error finding container 6825ed2e5df7fa22149e8d1111b464bd1ca6978608956be93c45db58a31dff14: Status 404 returned error can't find the container with id 6825ed2e5df7fa22149e8d1111b464bd1ca6978608956be93c45db58a31dff14
Oct 30 13:03:28.931141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.931108    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-8487b78549-kbm4t"]
Oct 30 13:03:28.934444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.934421    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:28.949244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.949196    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 13:03:28.949451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.949427    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 13:03:28.949536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.949452    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 13:03:28.949536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.949482    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7vwxh\""
Oct 30 13:03:28.949536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.949491    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 13:03:28.949536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.949531    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 13:03:28.949754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.949598    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-foph5sq9gg7nn\""
Oct 30 13:03:28.957337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:28.957311    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8487b78549-kbm4t"]
Oct 30 13:03:29.003515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.003482    4291 generic.go:356] "Generic (PLEG): container finished" podID="6df08611-a79d-4a59-9371-29dfa510b9be" containerID="101d37bac86d206496d198bb53a69caae0a1edb321ea6acbccee3ead6e2391ae" exitCode=0
Oct 30 13:03:29.003978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.003559    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdh7p" event={"ID":"6df08611-a79d-4a59-9371-29dfa510b9be","Type":"ContainerDied","Data":"101d37bac86d206496d198bb53a69caae0a1edb321ea6acbccee3ead6e2391ae"}
Oct 30 13:03:29.004911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.004890    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerStarted","Data":"6825ed2e5df7fa22149e8d1111b464bd1ca6978608956be93c45db58a31dff14"}
Oct 30 13:03:29.032838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.032778    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.032838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.032828    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.033007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.032903    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-tls\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.033007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.032937    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.033007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.032959    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmmb6\" (UniqueName: \"kubernetes.io/projected/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-kube-api-access-bmmb6\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.033135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.033078    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-metrics-client-ca\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.033135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.033111    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-tmp\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.033135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.033130    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.033268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.033147    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-grpc-tls\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.133902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.133876    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-metrics-client-ca\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.133917    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-tmp\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.133945    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134107    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-grpc-tls\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134217    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134286    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134348    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-tls\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134365    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-tmp\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134392    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134420    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmmb6\" (UniqueName: \"kubernetes.io/projected/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-kube-api-access-bmmb6\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.134722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.134670    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-metrics-client-ca\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.137438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.137397    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.137541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.137454    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.137959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.137937    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.138052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.137959    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.138684 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.138666    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-grpc-tls\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.139070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.139047    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-secret-thanos-querier-tls\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.156022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.155997    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmmb6\" (UniqueName: \"kubernetes.io/projected/8b3eb9b8-8fd3-474b-a3b9-2419e7e68704-kube-api-access-bmmb6\") pod \"thanos-querier-8487b78549-kbm4t\" (UID: \"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704\") " pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.252552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.252512    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:29.570427 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:29.570342    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-8487b78549-kbm4t"]
Oct 30 13:03:29.574992 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:29.574966    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b3eb9b8_8fd3_474b_a3b9_2419e7e68704.slice/crio-a1560977d729b01859f676a793ce234c99b3a64740220930d7c607c276545cd6 WatchSource:0}: Error finding container a1560977d729b01859f676a793ce234c99b3a64740220930d7c607c276545cd6: Status 404 returned error can't find the container with id a1560977d729b01859f676a793ce234c99b3a64740220930d7c607c276545cd6
Oct 30 13:03:30.012026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:30.011990    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" event={"ID":"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704","Type":"ContainerStarted","Data":"a1560977d729b01859f676a793ce234c99b3a64740220930d7c607c276545cd6"}
Oct 30 13:03:30.014514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:30.014487    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdh7p" event={"ID":"6df08611-a79d-4a59-9371-29dfa510b9be","Type":"ContainerStarted","Data":"609761cd6b3e9946dcfd8a62a12765c963004bed011c0658c8d8f4215bd34cf6"}
Oct 30 13:03:30.014655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:30.014544    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-pdh7p" event={"ID":"6df08611-a79d-4a59-9371-29dfa510b9be","Type":"ContainerStarted","Data":"c54130d88e32aee3b7a41eb0a7e04dc507caebedb0099075d50d4cde4102fa6c"}
Oct 30 13:03:30.016324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:30.016298    4291 generic.go:356] "Generic (PLEG): container finished" podID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerID="86d2712aec16395d463fb7e1a538dab53cd6d16f533faac085f597d864b8e225" exitCode=0
Oct 30 13:03:30.016404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:30.016349    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"86d2712aec16395d463fb7e1a538dab53cd6d16f533faac085f597d864b8e225"}
Oct 30 13:03:30.054580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:30.054527    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-pdh7p" podStartSLOduration=3.110939255 podStartE2EDuration="4.054509914s" totalImagesPullingTime="867.457341ms" totalInitContainerRuntime="76.113318ms" isStatefulPod=true podCreationTimestamp="2026-10-30 13:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:30.053926495 +0000 UTC m=+20.314164732" watchObservedRunningTime="2026-10-30 13:03:30.054509914 +0000 UTC m=+20.314748152"
Oct 30 13:03:30.978239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:30.978204    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-nsrbd"
Oct 30 13:03:32.027830 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.027792    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerStarted","Data":"e70f5c93f8817899025d28a1cefff06a00d00356065936ca297aa59d393b9b94"}
Oct 30 13:03:32.028246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.027859    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerStarted","Data":"d2f87c7439388dca78437ae71b52afe7391f3e1d8d0c06f8bb2ed966177a0641"}
Oct 30 13:03:32.028246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.027888    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerStarted","Data":"8d74a488d75e6914cbed465e755a00de9efc6d43971dae23ac8ba65912509bc5"}
Oct 30 13:03:32.028246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.027911    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerStarted","Data":"8947eae6ef7c63aa8b96bc58ce7bd4ce3ade0d80ad4f4eb59b309f48ca01e5f9"}
Oct 30 13:03:32.028246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.027939    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerStarted","Data":"f33ee5c935c68c3fd08d79fcde481d0a2bc373fa16068cd00c46b751c9990c99"}
Oct 30 13:03:32.030503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.030387    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" event={"ID":"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704","Type":"ContainerStarted","Data":"4fb7ddd7885ce90cdc971f310f3e011cdf94fa2808d0ed21e579ef24952fb441"}
Oct 30 13:03:32.030503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.030438    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" event={"ID":"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704","Type":"ContainerStarted","Data":"6452ce88471c53d734f1906fd1e514c6043ea2f0622c5ea56feb848666d8ba09"}
Oct 30 13:03:32.030503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:32.030463    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" event={"ID":"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704","Type":"ContainerStarted","Data":"8ea1399b9977a18aba3ed6d657f8e1ad0462687e7a3b8628aad4eb5fac3f157a"}
Oct 30 13:03:33.040046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.039972    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerStarted","Data":"e9e1a775f36fbbb5b39e694da47633643b81707d3491ee21078fad56187561a3"}
Oct 30 13:03:33.044009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.043980    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" event={"ID":"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704","Type":"ContainerStarted","Data":"799c30ba881ae58a4bb182820ee057a5ef00d740d22f2d005c63b8cd0e52d247"}
Oct 30 13:03:33.044141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.044052    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" event={"ID":"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704","Type":"ContainerStarted","Data":"ff86a78f2c28eef05dc69289adfe782441c85e9d650463693933013a06cdb958"}
Oct 30 13:03:33.044141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.044097    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" event={"ID":"8b3eb9b8-8fd3-474b-a3b9-2419e7e68704","Type":"ContainerStarted","Data":"78a3bfe25e523a1d4e356d23ca7f50052253c3004bfb5cd50593848e47b41706"}
Oct 30 13:03:33.044227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.044145    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:33.126433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.126381    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.767865789 podStartE2EDuration="5.12636642s" totalImagesPullingTime="3.341178582s" totalInitContainerRuntime="17.322049ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:28 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:33.113549694 +0000 UTC m=+23.373787928" watchObservedRunningTime="2026-10-30 13:03:33.12636642 +0000 UTC m=+23.386604653"
Oct 30 13:03:33.161589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.161534    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t" podStartSLOduration=2.2869185180000002 podStartE2EDuration="5.161512901s" totalImagesPullingTime="2.874594383s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:28 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:33.160406407 +0000 UTC m=+23.420644652" watchObservedRunningTime="2026-10-30 13:03:33.161512901 +0000 UTC m=+23.421751137"
Oct 30 13:03:33.691570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.691542    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:03:33.708087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.708059    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.723721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.723692    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 13:03:33.723893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.723875    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 13:03:33.724037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.723975    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:03:33.747318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747286    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 13:03:33.747460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747329    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 13:03:33.747460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747300    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 13:03:33.747460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747286    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 13:03:33.747460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747422    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 13:03:33.747460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747447    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 13:03:33.747607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747546    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5jvh5\""
Oct 30 13:03:33.747607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747585    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-17u4oko5qgjlf\""
Oct 30 13:03:33.747739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747723    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 13:03:33.747917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.747897    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 13:03:33.748100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.748082    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 13:03:33.824054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.824026    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 13:03:33.824345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.824327    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 13:03:33.837940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.837915    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 13:03:33.838063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.838045    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 13:03:33.880332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880302    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880339    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-config\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880359    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880416    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880526    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880552    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880581    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxp4v\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-kube-api-access-wxp4v\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880597    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880674    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880769    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-config-out\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880802    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880869    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880893    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-web-config\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.880992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.880962    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.881071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.881012    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.881071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.881054    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.881141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.881096    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.881141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.881127    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.881228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.881213    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.881269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.881238    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982338    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982386    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-config\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982512    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982544    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982589    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982615    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982653    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxp4v\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-kube-api-access-wxp4v\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982694    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982712    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982740    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-config-out\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982762    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982790    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982809    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-web-config\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982846    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982882    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.982923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982906    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.983346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982929    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.983346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982948    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.983346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.982979    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.983346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.983006    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.983346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.983072    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.983677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.983556    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.984048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.984016    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.985658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.985613    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.985782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.985691    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.986910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.985954    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.986910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.986552    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.987899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.987313    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.987899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.987448    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.987899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.987852    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.988567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.988541    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.989105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.989081    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-config-out\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.989217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.989090    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.989751 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.989726    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.989839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.989790    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.989839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.989802    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-web-config\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.990719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.990606    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.991376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.991354    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:33.992042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:33.992025    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-config\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:34.005768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:34.005751    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxp4v\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-kube-api-access-wxp4v\") pod \"prometheus-k8s-0\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:34.073532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:34.073499    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:34.444001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:34.443970    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:03:34.461854 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:03:34.461814    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod177d2f52_bf23_4d17_96e8_6f6af0223f07.slice/crio-2be188f89a804565d43819b1077c4026301a3330522655c84122eab6cb5ecb6b WatchSource:0}: Error finding container 2be188f89a804565d43819b1077c4026301a3330522655c84122eab6cb5ecb6b: Status 404 returned error can't find the container with id 2be188f89a804565d43819b1077c4026301a3330522655c84122eab6cb5ecb6b
Oct 30 13:03:35.052514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:35.052485    4291 generic.go:356] "Generic (PLEG): container finished" podID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57" exitCode=0
Oct 30 13:03:35.052697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:35.052526    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"}
Oct 30 13:03:35.052697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:35.052575    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerStarted","Data":"2be188f89a804565d43819b1077c4026301a3330522655c84122eab6cb5ecb6b"}
Oct 30 13:03:36.071981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:36.071921    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-rmscn" event={"ID":"c18f5676-dc4b-45af-a3af-af8db22da5b2","Type":"ContainerStarted","Data":"387ff67f8a23689c3949d8e1d9e84a02e4c6edf6a714cd22f9fccc9af7988b24"}
Oct 30 13:03:36.072698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:36.072666    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:36.083770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:36.083706    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-rmscn"
Oct 30 13:03:36.113760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:36.113710    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-rmscn" podStartSLOduration=2.355582029 podStartE2EDuration="20.113697397s" totalImagesPullingTime="17.758115368s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:36.113261296 +0000 UTC m=+26.373499549" watchObservedRunningTime="2026-10-30 13:03:36.113697397 +0000 UTC m=+26.373935628"
Oct 30 13:03:38.216601 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:03:38.216563    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:03:38.273908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:38.273877    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8542k"
Oct 30 13:03:39.054444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:39.054416    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-8487b78549-kbm4t"
Oct 30 13:03:40.054148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.054117    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7964b8d7bd-cpfkm"
Oct 30 13:03:40.093208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.093164    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerStarted","Data":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"}
Oct 30 13:03:40.093364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.093253    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerStarted","Data":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"}
Oct 30 13:03:40.093364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.093287    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerStarted","Data":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"}
Oct 30 13:03:40.093364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.093333    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerStarted","Data":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"}
Oct 30 13:03:40.093364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.093363    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerStarted","Data":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"}
Oct 30 13:03:40.093547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.093402    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerStarted","Data":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"}
Oct 30 13:03:40.149238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:40.149078    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.067554078 podStartE2EDuration="7.149057042s" totalImagesPullingTime="4.057861635s" totalInitContainerRuntime="23.641329ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:03:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:03:40.147121285 +0000 UTC m=+30.407359589" watchObservedRunningTime="2026-10-30 13:03:40.149057042 +0000 UTC m=+30.409295291"
Oct 30 13:03:42.720429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:42.720385    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:42.774004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:42.773976    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:43.106881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:43.106848    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:03:43.124119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:03:43.124093    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:08.211226 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:08.211187    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:04:13.508975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:13.508941    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 13:04:13.509429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:13.509374    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-web" containerID="cri-o://8d74a488d75e6914cbed465e755a00de9efc6d43971dae23ac8ba65912509bc5" gracePeriod=120
Oct 30 13:04:13.509569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:13.509434    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="config-reloader" containerID="cri-o://8947eae6ef7c63aa8b96bc58ce7bd4ce3ade0d80ad4f4eb59b309f48ca01e5f9" gracePeriod=120
Oct 30 13:04:13.509569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:13.509372    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-metric" containerID="cri-o://e70f5c93f8817899025d28a1cefff06a00d00356065936ca297aa59d393b9b94" gracePeriod=120
Oct 30 13:04:13.509569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:13.509357    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="alertmanager" containerID="cri-o://f33ee5c935c68c3fd08d79fcde481d0a2bc373fa16068cd00c46b751c9990c99" gracePeriod=120
Oct 30 13:04:13.509569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:13.509430    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy" containerID="cri-o://d2f87c7439388dca78437ae71b52afe7391f3e1d8d0c06f8bb2ed966177a0641" gracePeriod=120
Oct 30 13:04:13.509832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:13.509445    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="prom-label-proxy" containerID="cri-o://e9e1a775f36fbbb5b39e694da47633643b81707d3491ee21078fad56187561a3" gracePeriod=120
Oct 30 13:04:14.222085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222054    4291 generic.go:356] "Generic (PLEG): container finished" podID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerID="e9e1a775f36fbbb5b39e694da47633643b81707d3491ee21078fad56187561a3" exitCode=0
Oct 30 13:04:14.222085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222081    4291 generic.go:356] "Generic (PLEG): container finished" podID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerID="d2f87c7439388dca78437ae71b52afe7391f3e1d8d0c06f8bb2ed966177a0641" exitCode=0
Oct 30 13:04:14.222085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222092    4291 generic.go:356] "Generic (PLEG): container finished" podID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerID="8947eae6ef7c63aa8b96bc58ce7bd4ce3ade0d80ad4f4eb59b309f48ca01e5f9" exitCode=0
Oct 30 13:04:14.222085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222101    4291 generic.go:356] "Generic (PLEG): container finished" podID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerID="f33ee5c935c68c3fd08d79fcde481d0a2bc373fa16068cd00c46b751c9990c99" exitCode=0
Oct 30 13:04:14.222423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222135    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"e9e1a775f36fbbb5b39e694da47633643b81707d3491ee21078fad56187561a3"}
Oct 30 13:04:14.222423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222201    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"d2f87c7439388dca78437ae71b52afe7391f3e1d8d0c06f8bb2ed966177a0641"}
Oct 30 13:04:14.222423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222227    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"8947eae6ef7c63aa8b96bc58ce7bd4ce3ade0d80ad4f4eb59b309f48ca01e5f9"}
Oct 30 13:04:14.222423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:14.222252    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"f33ee5c935c68c3fd08d79fcde481d0a2bc373fa16068cd00c46b751c9990c99"}
Oct 30 13:04:15.230047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.230013    4291 generic.go:356] "Generic (PLEG): container finished" podID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerID="e70f5c93f8817899025d28a1cefff06a00d00356065936ca297aa59d393b9b94" exitCode=0
Oct 30 13:04:15.230047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.230043    4291 generic.go:356] "Generic (PLEG): container finished" podID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerID="8d74a488d75e6914cbed465e755a00de9efc6d43971dae23ac8ba65912509bc5" exitCode=0
Oct 30 13:04:15.230462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.230078    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"e70f5c93f8817899025d28a1cefff06a00d00356065936ca297aa59d393b9b94"}
Oct 30 13:04:15.230462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.230141    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"8d74a488d75e6914cbed465e755a00de9efc6d43971dae23ac8ba65912509bc5"}
Oct 30 13:04:15.275064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.275038    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:04:15.425518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425446    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425491    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-out\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425512    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-tls-assets\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425535    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425581    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-kube-api-access-zsx4x\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-kube-api-access-zsx4x\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425599    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-volume\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-volume\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425630    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-web-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-web-config\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425695    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-cluster-tls-config\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425757    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-trusted-ca-bundle\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.425808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425785    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-metrics-client-ca\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.426220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425812    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-main-db\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.426220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425844    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-main-tls\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.426220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.425878    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-web\") pod \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\" (UID: \"b2a87588-274a-4c9c-89a6-3fa864fc18ec\") "
Oct 30 13:04:15.426385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.426357    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-trusted-ca-bundle" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:15.426853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.426592    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-main-db" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:15.426853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.426678    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-metrics-client-ca" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:15.428707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.428654    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-metric" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:15.428804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.428748    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-tls-assets" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:15.428872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.428813    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-kube-api-access-zsx4x" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "kube-api-access-zsx4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:15.428929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.428900    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-out" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:15.429307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.429261    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:15.429398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.429304    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-volume" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:15.429651 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.429611    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-main-tls" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:15.429837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.429818    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-web" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:15.434425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.434385    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-cluster-tls-config" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:15.440327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.440304    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-web-config" pod "b2a87588-274a-4c9c-89a6-3fa864fc18ec" (UID: "b2a87588-274a-4c9c-89a6-3fa864fc18ec"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:15.527000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.526972    4291 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.526994    4291 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-out\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527003    4291 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-tls-assets\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527012    4291 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527021    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsx4x\" (UniqueName: \"kubernetes.io/projected/b2a87588-274a-4c9c-89a6-3fa864fc18ec-kube-api-access-zsx4x\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527031    4291 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527041    4291 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-web-config\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527051    4291 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-cluster-tls-config\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527058    4291 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527067    4291 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2a87588-274a-4c9c-89a6-3fa864fc18ec-metrics-client-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527075    4291 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b2a87588-274a-4c9c-89a6-3fa864fc18ec-alertmanager-main-db\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527083    4291 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-main-tls\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:15.527183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:15.527092    4291 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b2a87588-274a-4c9c-89a6-3fa864fc18ec-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:16.236721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.236619    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b2a87588-274a-4c9c-89a6-3fa864fc18ec","Type":"ContainerDied","Data":"6825ed2e5df7fa22149e8d1111b464bd1ca6978608956be93c45db58a31dff14"}
Oct 30 13:04:16.236721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.236678    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 13:04:16.237168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.236735    4291 scope.go:122] "RemoveContainer" containerID="e9e1a775f36fbbb5b39e694da47633643b81707d3491ee21078fad56187561a3"
Oct 30 13:04:16.244200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.244181    4291 scope.go:122] "RemoveContainer" containerID="e70f5c93f8817899025d28a1cefff06a00d00356065936ca297aa59d393b9b94"
Oct 30 13:04:16.250710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.250693    4291 scope.go:122] "RemoveContainer" containerID="d2f87c7439388dca78437ae71b52afe7391f3e1d8d0c06f8bb2ed966177a0641"
Oct 30 13:04:16.257008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.256991    4291 scope.go:122] "RemoveContainer" containerID="8d74a488d75e6914cbed465e755a00de9efc6d43971dae23ac8ba65912509bc5"
Oct 30 13:04:16.263086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.263065    4291 scope.go:122] "RemoveContainer" containerID="8947eae6ef7c63aa8b96bc58ce7bd4ce3ade0d80ad4f4eb59b309f48ca01e5f9"
Oct 30 13:04:16.269702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.269678    4291 scope.go:122] "RemoveContainer" containerID="f33ee5c935c68c3fd08d79fcde481d0a2bc373fa16068cd00c46b751c9990c99"
Oct 30 13:04:16.276195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.276179    4291 scope.go:122] "RemoveContainer" containerID="86d2712aec16395d463fb7e1a538dab53cd6d16f533faac085f597d864b8e225"
Oct 30 13:04:16.301186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.301151    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 13:04:16.312925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:16.312900    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 13:04:17.876067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:17.876032    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" path="/var/lib/kubelet/pods/b2a87588-274a-4c9c-89a6-3fa864fc18ec/volumes"
Oct 30 13:04:18.151796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.148700    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:04:18.152297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.152261    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="prometheus" containerID="cri-o://7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115" gracePeriod=600
Oct 30 13:04:18.152473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.152268    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="thanos-sidecar" containerID="cri-o://e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11" gracePeriod=600
Oct 30 13:04:18.152572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.152299    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-web" containerID="cri-o://1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f" gracePeriod=600
Oct 30 13:04:18.152658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.152317    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="config-reloader" containerID="cri-o://a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f" gracePeriod=600
Oct 30 13:04:18.152709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.152311    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy" containerID="cri-o://d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be" gracePeriod=600
Oct 30 13:04:18.152769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.152336    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-thanos" containerID="cri-o://6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35" gracePeriod=600
Oct 30 13:04:18.426057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.426001    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:18.549912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.549877    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-config\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.549916    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.549960    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-metrics-client-certs\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550011    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-web-config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-web-config\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550042    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-db\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550074    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-2\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550110    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-kubelet-serving-ca-bundle\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550141    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-kube-rbac-proxy\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550168    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-1\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550206    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550246    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-kube-api-access-wxp4v\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-kube-api-access-wxp4v\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550278    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-tls\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550312    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-grpc-tls\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550344    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-trusted-ca-bundle\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550388    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-metrics-client-ca\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550426    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-0\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550426    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-2" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550459    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-config-out\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-config-out\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550493    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-thanos-prometheus-http-client-file\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550525    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-serving-certs-ca-bundle\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550559    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-tls-assets\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-tls-assets\") pod \"177d2f52-bf23-4d17-96e8-6f6af0223f07\" (UID: \"177d2f52-bf23-4d17-96e8-6f6af0223f07\") "
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550799    4291 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.550857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.550838    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-kubelet-serving-ca-bundle" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:18.551260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.551171    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-1" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:18.551773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.551364    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-db" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:18.552627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.552372    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-trusted-ca-bundle" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:18.553573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.553028    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-0" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:18.553573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.553118    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-tls-assets" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:18.553573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.553321    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-config" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.553573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.553457    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-metrics-client-ca" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:18.553862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.553747    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-metrics-client-certs" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.554062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.554025    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-serving-certs-ca-bundle" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:04:18.554604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.554566    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-kube-rbac-proxy" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.554868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.554843    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-grpc-tls" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.554868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.554843    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-kube-api-access-wxp4v" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "kube-api-access-wxp4v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:04:18.555008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.554962    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-thanos-prometheus-http-client-file" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.555208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.555187    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-kube-rbac-proxy-web" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.555430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.555415    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-config-out" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:04:18.555733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.555700    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-tls" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.556226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.556210    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-thanos-sidecar-tls" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.562779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.562759    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-web-config" pod "177d2f52-bf23-4d17-96e8-6f6af0223f07" (UID: "177d2f52-bf23-4d17-96e8-6f6af0223f07"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:04:18.651292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651258    4291 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651287    4291 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-kube-rbac-proxy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651297    4291 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651309    4291 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651319    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxp4v\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-kube-api-access-wxp4v\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651327    4291 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-tls\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651336    4291 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-grpc-tls\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651345    4291 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-trusted-ca-bundle\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651353    4291 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-metrics-client-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651361    4291 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651369    4291 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-config-out\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651377    4291 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-thanos-prometheus-http-client-file\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651386    4291 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/177d2f52-bf23-4d17-96e8-6f6af0223f07-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651394    4291 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/177d2f52-bf23-4d17-96e8-6f6af0223f07-tls-assets\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651402    4291 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-config\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651410    4291 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651417    4291 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-secret-metrics-client-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651425    4291 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/177d2f52-bf23-4d17-96e8-6f6af0223f07-web-config\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:18.651484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:18.651433    4291 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/177d2f52-bf23-4d17-96e8-6f6af0223f07-prometheus-k8s-db\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:04:19.250686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250630    4291 generic.go:356] "Generic (PLEG): container finished" podID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35" exitCode=0
Oct 30 13:04:19.250686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250664    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"}
Oct 30 13:04:19.250686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250679    4291 generic.go:356] "Generic (PLEG): container finished" podID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be" exitCode=0
Oct 30 13:04:19.250686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250690    4291 generic.go:356] "Generic (PLEG): container finished" podID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f" exitCode=0
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250699    4291 generic.go:356] "Generic (PLEG): container finished" podID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11" exitCode=0
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250707    4291 generic.go:356] "Generic (PLEG): container finished" podID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f" exitCode=0
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250715    4291 generic.go:356] "Generic (PLEG): container finished" podID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115" exitCode=0
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250724    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250746    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"}
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250786    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"}
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250813    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"}
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250815    4291 scope.go:122] "RemoveContainer" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250837    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"}
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250871    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"}
Oct 30 13:04:19.251153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.250897    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"177d2f52-bf23-4d17-96e8-6f6af0223f07","Type":"ContainerDied","Data":"2be188f89a804565d43819b1077c4026301a3330522655c84122eab6cb5ecb6b"}
Oct 30 13:04:19.258455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.258398    4291 scope.go:122] "RemoveContainer" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.265007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.264989    4291 scope.go:122] "RemoveContainer" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.271126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.271108    4291 scope.go:122] "RemoveContainer" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.277067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.277051    4291 scope.go:122] "RemoveContainer" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.283277 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.283258    4291 scope.go:122] "RemoveContainer" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.289951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.289931    4291 scope.go:122] "RemoveContainer" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.295956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.295940    4291 scope.go:122] "RemoveContainer" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.296226 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:19.296207    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": container with ID starting with 6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35 not found: ID does not exist" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.296274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.296237    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"} err="failed to get container status \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": rpc error: code = NotFound desc = could not find container \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": container with ID starting with 6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35 not found: ID does not exist"
Oct 30 13:04:19.296274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.296267    4291 scope.go:122] "RemoveContainer" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.296500 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:19.296483    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": container with ID starting with d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be not found: ID does not exist" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.296540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.296506    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"} err="failed to get container status \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": rpc error: code = NotFound desc = could not find container \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": container with ID starting with d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be not found: ID does not exist"
Oct 30 13:04:19.296540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.296522    4291 scope.go:122] "RemoveContainer" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.296804 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:19.296789    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": container with ID starting with 1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f not found: ID does not exist" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.296849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.296807    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"} err="failed to get container status \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": rpc error: code = NotFound desc = could not find container \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": container with ID starting with 1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f not found: ID does not exist"
Oct 30 13:04:19.296849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.296820    4291 scope.go:122] "RemoveContainer" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.297041 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:19.297026    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": container with ID starting with e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11 not found: ID does not exist" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.297079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297046    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"} err="failed to get container status \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": rpc error: code = NotFound desc = could not find container \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": container with ID starting with e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11 not found: ID does not exist"
Oct 30 13:04:19.297079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297059    4291 scope.go:122] "RemoveContainer" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.297262 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:19.297249    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": container with ID starting with a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f not found: ID does not exist" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.297311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297266    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"} err="failed to get container status \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": rpc error: code = NotFound desc = could not find container \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": container with ID starting with a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f not found: ID does not exist"
Oct 30 13:04:19.297311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297276    4291 scope.go:122] "RemoveContainer" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.297485 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:19.297469    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": container with ID starting with 7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115 not found: ID does not exist" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.297523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297488    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"} err="failed to get container status \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": rpc error: code = NotFound desc = could not find container \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": container with ID starting with 7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115 not found: ID does not exist"
Oct 30 13:04:19.297523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297501    4291 scope.go:122] "RemoveContainer" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.297791 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:19.297775    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": container with ID starting with a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57 not found: ID does not exist" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.297851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297791    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"} err="failed to get container status \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": rpc error: code = NotFound desc = could not find container \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": container with ID starting with a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57 not found: ID does not exist"
Oct 30 13:04:19.297851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.297803    4291 scope.go:122] "RemoveContainer" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.298029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298009    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"} err="failed to get container status \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": rpc error: code = NotFound desc = could not find container \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": container with ID starting with 6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35 not found: ID does not exist"
Oct 30 13:04:19.298091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298031    4291 scope.go:122] "RemoveContainer" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.298267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298249    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"} err="failed to get container status \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": rpc error: code = NotFound desc = could not find container \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": container with ID starting with d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be not found: ID does not exist"
Oct 30 13:04:19.298313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298267    4291 scope.go:122] "RemoveContainer" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.298463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298446    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"} err="failed to get container status \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": rpc error: code = NotFound desc = could not find container \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": container with ID starting with 1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f not found: ID does not exist"
Oct 30 13:04:19.298529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298464    4291 scope.go:122] "RemoveContainer" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.298731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298715    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"} err="failed to get container status \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": rpc error: code = NotFound desc = could not find container \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": container with ID starting with e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11 not found: ID does not exist"
Oct 30 13:04:19.298784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298731    4291 scope.go:122] "RemoveContainer" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.298933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298915    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"} err="failed to get container status \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": rpc error: code = NotFound desc = could not find container \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": container with ID starting with a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f not found: ID does not exist"
Oct 30 13:04:19.298973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.298934    4291 scope.go:122] "RemoveContainer" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.299132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299117    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"} err="failed to get container status \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": rpc error: code = NotFound desc = could not find container \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": container with ID starting with 7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115 not found: ID does not exist"
Oct 30 13:04:19.299170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299132    4291 scope.go:122] "RemoveContainer" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.299324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299307    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"} err="failed to get container status \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": rpc error: code = NotFound desc = could not find container \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": container with ID starting with a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57 not found: ID does not exist"
Oct 30 13:04:19.299370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299325    4291 scope.go:122] "RemoveContainer" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.299501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299485    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"} err="failed to get container status \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": rpc error: code = NotFound desc = could not find container \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": container with ID starting with 6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35 not found: ID does not exist"
Oct 30 13:04:19.299541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299503    4291 scope.go:122] "RemoveContainer" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.299710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299693    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"} err="failed to get container status \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": rpc error: code = NotFound desc = could not find container \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": container with ID starting with d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be not found: ID does not exist"
Oct 30 13:04:19.299758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299710    4291 scope.go:122] "RemoveContainer" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.299889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299871    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"} err="failed to get container status \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": rpc error: code = NotFound desc = could not find container \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": container with ID starting with 1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f not found: ID does not exist"
Oct 30 13:04:19.299889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.299887    4291 scope.go:122] "RemoveContainer" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.300098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300082    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"} err="failed to get container status \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": rpc error: code = NotFound desc = could not find container \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": container with ID starting with e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11 not found: ID does not exist"
Oct 30 13:04:19.300098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300098    4291 scope.go:122] "RemoveContainer" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.300291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300274    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"} err="failed to get container status \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": rpc error: code = NotFound desc = could not find container \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": container with ID starting with a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f not found: ID does not exist"
Oct 30 13:04:19.300291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300288    4291 scope.go:122] "RemoveContainer" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.300493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300475    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"} err="failed to get container status \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": rpc error: code = NotFound desc = could not find container \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": container with ID starting with 7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115 not found: ID does not exist"
Oct 30 13:04:19.300493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300491    4291 scope.go:122] "RemoveContainer" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.300713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300697    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"} err="failed to get container status \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": rpc error: code = NotFound desc = could not find container \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": container with ID starting with a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57 not found: ID does not exist"
Oct 30 13:04:19.300713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300712    4291 scope.go:122] "RemoveContainer" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.300916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300898    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"} err="failed to get container status \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": rpc error: code = NotFound desc = could not find container \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": container with ID starting with 6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35 not found: ID does not exist"
Oct 30 13:04:19.300969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.300916    4291 scope.go:122] "RemoveContainer" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.301128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301110    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"} err="failed to get container status \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": rpc error: code = NotFound desc = could not find container \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": container with ID starting with d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be not found: ID does not exist"
Oct 30 13:04:19.301176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301130    4291 scope.go:122] "RemoveContainer" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.301351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301336    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"} err="failed to get container status \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": rpc error: code = NotFound desc = could not find container \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": container with ID starting with 1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f not found: ID does not exist"
Oct 30 13:04:19.301351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301349    4291 scope.go:122] "RemoveContainer" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.301543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301527    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"} err="failed to get container status \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": rpc error: code = NotFound desc = could not find container \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": container with ID starting with e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11 not found: ID does not exist"
Oct 30 13:04:19.301580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301542    4291 scope.go:122] "RemoveContainer" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.301757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301743    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"} err="failed to get container status \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": rpc error: code = NotFound desc = could not find container \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": container with ID starting with a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f not found: ID does not exist"
Oct 30 13:04:19.301757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301756    4291 scope.go:122] "RemoveContainer" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.301958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301940    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"} err="failed to get container status \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": rpc error: code = NotFound desc = could not find container \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": container with ID starting with 7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115 not found: ID does not exist"
Oct 30 13:04:19.301998 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.301958    4291 scope.go:122] "RemoveContainer" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.302159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302142    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"} err="failed to get container status \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": rpc error: code = NotFound desc = could not find container \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": container with ID starting with a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57 not found: ID does not exist"
Oct 30 13:04:19.302199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302160    4291 scope.go:122] "RemoveContainer" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.302371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302355    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"} err="failed to get container status \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": rpc error: code = NotFound desc = could not find container \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": container with ID starting with 6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35 not found: ID does not exist"
Oct 30 13:04:19.302414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302371    4291 scope.go:122] "RemoveContainer" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.302587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302568    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"} err="failed to get container status \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": rpc error: code = NotFound desc = could not find container \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": container with ID starting with d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be not found: ID does not exist"
Oct 30 13:04:19.302626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302587    4291 scope.go:122] "RemoveContainer" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.302811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302791    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"} err="failed to get container status \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": rpc error: code = NotFound desc = could not find container \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": container with ID starting with 1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f not found: ID does not exist"
Oct 30 13:04:19.302888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.302812    4291 scope.go:122] "RemoveContainer" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.303048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303031    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"} err="failed to get container status \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": rpc error: code = NotFound desc = could not find container \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": container with ID starting with e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11 not found: ID does not exist"
Oct 30 13:04:19.303094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303049    4291 scope.go:122] "RemoveContainer" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.303242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303223    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"} err="failed to get container status \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": rpc error: code = NotFound desc = could not find container \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": container with ID starting with a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f not found: ID does not exist"
Oct 30 13:04:19.303308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303242    4291 scope.go:122] "RemoveContainer" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.303447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303430    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"} err="failed to get container status \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": rpc error: code = NotFound desc = could not find container \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": container with ID starting with 7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115 not found: ID does not exist"
Oct 30 13:04:19.303494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303448    4291 scope.go:122] "RemoveContainer" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.303665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303647    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"} err="failed to get container status \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": rpc error: code = NotFound desc = could not find container \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": container with ID starting with a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57 not found: ID does not exist"
Oct 30 13:04:19.303736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303666    4291 scope.go:122] "RemoveContainer" containerID="6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"
Oct 30 13:04:19.303875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303859    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35"} err="failed to get container status \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": rpc error: code = NotFound desc = could not find container \"6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35\": container with ID starting with 6e6898bf48de751ff3f7fae0298aadb767c11a9c65abae08b646302e76592e35 not found: ID does not exist"
Oct 30 13:04:19.303916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.303876    4291 scope.go:122] "RemoveContainer" containerID="d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"
Oct 30 13:04:19.304061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304042    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be"} err="failed to get container status \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": rpc error: code = NotFound desc = could not find container \"d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be\": container with ID starting with d70bc3b90db112ee73f6f8fc58eb3f68ee3e22b90583ddcdb73c9007c749b2be not found: ID does not exist"
Oct 30 13:04:19.304108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304062    4291 scope.go:122] "RemoveContainer" containerID="1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"
Oct 30 13:04:19.304249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304232    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f"} err="failed to get container status \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": rpc error: code = NotFound desc = could not find container \"1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f\": container with ID starting with 1fbf7d87f24a2064ec5ec7aab95ee5e63e5a7a428461896dd49a0b00ac17566f not found: ID does not exist"
Oct 30 13:04:19.304315 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304250    4291 scope.go:122] "RemoveContainer" containerID="e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"
Oct 30 13:04:19.304445 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304421    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11"} err="failed to get container status \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": rpc error: code = NotFound desc = could not find container \"e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11\": container with ID starting with e50deeabfbee7a228ec2681a7219157392ef6b71418e9be079039075a59edf11 not found: ID does not exist"
Oct 30 13:04:19.304445 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304440    4291 scope.go:122] "RemoveContainer" containerID="a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"
Oct 30 13:04:19.304625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304608    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f"} err="failed to get container status \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": rpc error: code = NotFound desc = could not find container \"a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f\": container with ID starting with a6f9453d537fca02a04a40bfb5dd65e9f9326670ebe87c8749531fd51f3eaa5f not found: ID does not exist"
Oct 30 13:04:19.304705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304627    4291 scope.go:122] "RemoveContainer" containerID="7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"
Oct 30 13:04:19.304831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304816    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115"} err="failed to get container status \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": rpc error: code = NotFound desc = could not find container \"7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115\": container with ID starting with 7581908ccf829edd23ecc6d734506523d0fe858dc32bf44e6286955b12345115 not found: ID does not exist"
Oct 30 13:04:19.304881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.304831    4291 scope.go:122] "RemoveContainer" containerID="a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"
Oct 30 13:04:19.305016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.305001    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57"} err="failed to get container status \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": rpc error: code = NotFound desc = could not find container \"a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57\": container with ID starting with a3aee8db3fad7260e447c8c0256625f68a2140717b7d5ad9b9ab67b754932b57 not found: ID does not exist"
Oct 30 13:04:19.316130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.316112    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:04:19.328157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.328140    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:04:19.347305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347287    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:04:19.347529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347517    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="config-reloader"
Oct 30 13:04:19.347574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347530    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="config-reloader"
Oct 30 13:04:19.347574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347546    4291 cpu_manager.go:414] "containerMap: removing container" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-thanos"
Oct 30 13:04:19.347574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347551    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-thanos"
Oct 30 13:04:19.347574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347562    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="alertmanager"
Oct 30 13:04:19.347574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347566    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="alertmanager"
Oct 30 13:04:19.347574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347573    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy"
Oct 30 13:04:19.347574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347577    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347583    4291 cpu_manager.go:414] "containerMap: removing container" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="init-config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347588    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="init-config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347595    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="prom-label-proxy"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347600    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="prom-label-proxy"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347608    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="init-config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347612    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="init-config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347619    4291 cpu_manager.go:414] "containerMap: removing container" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="thanos-sidecar"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347623    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="thanos-sidecar"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347630    4291 cpu_manager.go:414] "containerMap: removing container" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="prometheus"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347649    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="prometheus"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347655    4291 cpu_manager.go:414] "containerMap: removing container" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-web"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347659    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-web"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347670    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-metric"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347674    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-metric"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347680    4291 cpu_manager.go:414] "containerMap: removing container" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347697    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347704    4291 cpu_manager.go:414] "containerMap: removing container" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347710    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347717    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-web"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347721    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-web"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347759    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-metric"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347767    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347797    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347802    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy-web"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347807    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="config-reloader"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347814    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-thanos"
Oct 30 13:04:19.347810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347820    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="alertmanager"
Oct 30 13:04:19.348621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347826    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="kube-rbac-proxy"
Oct 30 13:04:19.348621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347831    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2a87588-274a-4c9c-89a6-3fa864fc18ec" containerName="prom-label-proxy"
Oct 30 13:04:19.348621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347836    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="thanos-sidecar"
Oct 30 13:04:19.348621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347841    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="prometheus"
Oct 30 13:04:19.348621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.347845    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" containerName="kube-rbac-proxy-web"
Oct 30 13:04:19.366695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.366673    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.375160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.375139    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:04:19.381520 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.381503    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 13:04:19.381605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.381565    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 13:04:19.381755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.381740    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 13:04:19.381888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.381876    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 13:04:19.382279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382261    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 13:04:19.382279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382274    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 13:04:19.382418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382274    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 13:04:19.382418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382290    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 13:04:19.382418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382365    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 13:04:19.382418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382377    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 13:04:19.382616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382485    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 13:04:19.382616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382506    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 13:04:19.382616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382523    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 13:04:19.382616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382523    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-5jvh5\""
Oct 30 13:04:19.382616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.382578    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-17u4oko5qgjlf\""
Oct 30 13:04:19.395270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.395252    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 13:04:19.395270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.395264    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 13:04:19.457536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457484    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a418318-e714-4cad-beda-5aca5d648753-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457525    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-config\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457552    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a418318-e714-4cad-beda-5aca5d648753-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457658    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457701    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457717    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457733    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457751    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457768    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457867    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457895    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.457917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457912    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpxcq\" (UniqueName: \"kubernetes.io/projected/3a418318-e714-4cad-beda-5aca5d648753-kube-api-access-cpxcq\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457929    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457965    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.457996    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.458031    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.458055    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.458092    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.458134    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.458208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.458153    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559217    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559247    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559264    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559285    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559309    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559336    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559366    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559394    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559419    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpxcq\" (UniqueName: \"kubernetes.io/projected/3a418318-e714-4cad-beda-5aca5d648753-kube-api-access-cpxcq\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559445    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559470    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559495    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559523    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559548    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559582    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559611    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559657    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559695    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a418318-e714-4cad-beda-5aca5d648753-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559745    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-config\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559779    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a418318-e714-4cad-beda-5aca5d648753-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.559973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.559889    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.560334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.560038    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.560334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.560201    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.560334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.560263    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.560893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.560870    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.562467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.562441    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3a418318-e714-4cad-beda-5aca5d648753-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.562577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.562469    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.563256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.562923    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.563256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.562948    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-web-config\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.563256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.563171    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.563256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.563191    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.563791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.563710    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.564110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.564083    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.564188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.564085    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3a418318-e714-4cad-beda-5aca5d648753-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.564613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.564590    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3a418318-e714-4cad-beda-5aca5d648753-config-out\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.564720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.564626    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.565354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.565332    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.565480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.565462    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.565582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.565560    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3a418318-e714-4cad-beda-5aca5d648753-config\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.581081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.581058    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpxcq\" (UniqueName: \"kubernetes.io/projected/3a418318-e714-4cad-beda-5aca5d648753-kube-api-access-cpxcq\") pod \"prometheus-k8s-0\" (UID: \"3a418318-e714-4cad-beda-5aca5d648753\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.743871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.743788    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:19.875782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:19.875753    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="177d2f52-bf23-4d17-96e8-6f6af0223f07" path="/var/lib/kubelet/pods/177d2f52-bf23-4d17-96e8-6f6af0223f07/volumes"
Oct 30 13:04:20.093979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:20.093953    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 13:04:20.256593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:20.256559    4291 generic.go:356] "Generic (PLEG): container finished" podID="3a418318-e714-4cad-beda-5aca5d648753" containerID="672792a48d98cc925cea3fda50ea0ebf4511ebc5b5e8eade49014080de7ebded" exitCode=0
Oct 30 13:04:20.256922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:20.256609    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerDied","Data":"672792a48d98cc925cea3fda50ea0ebf4511ebc5b5e8eade49014080de7ebded"}
Oct 30 13:04:20.256922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:20.256723    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerStarted","Data":"83404c74ee77b7deb52aa8914994435557c3f85de7a55ccdd6a95d6295fccbe1"}
Oct 30 13:04:21.263515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:21.263483    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerStarted","Data":"dd11bd9170672e4b65251d17211b2ee89287f207041e1d86ee17954b38559d98"}
Oct 30 13:04:21.263923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:21.263545    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerStarted","Data":"50f6598676d58ebee9f8577152b9030c1d1075e64d1380be2e8c7b958fd90a53"}
Oct 30 13:04:21.263923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:21.263571    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerStarted","Data":"bc66b6df84e3418c658de3cde40703eca808574d2926c037d03c729f862fceec"}
Oct 30 13:04:21.263923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:21.263597    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerStarted","Data":"684e49dd64392ec31c1c6cbbb52585fcd01022a038fd930b4e1ccbf866e5f3b3"}
Oct 30 13:04:21.263923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:21.263651    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerStarted","Data":"efb98893a2a95603dfc49d86fae8c12e5f0de9162fda01a94731724b68ece6d7"}
Oct 30 13:04:21.263923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:21.263678    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3a418318-e714-4cad-beda-5aca5d648753","Type":"ContainerStarted","Data":"d253b9a149e83ff95220b39bead10e379f19727982d48f95e9df12e0c6bef55d"}
Oct 30 13:04:21.327567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:21.327510    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.297942179 podStartE2EDuration="2.327498086s" totalImagesPullingTime="0s" totalInitContainerRuntime="29.555907ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:04:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:04:21.326868549 +0000 UTC m=+71.587106783" watchObservedRunningTime="2026-10-30 13:04:21.327498086 +0000 UTC m=+71.587736320"
Oct 30 13:04:24.744850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:04:24.744788    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:04:38.208356 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:04:38.208313    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:05:08.208201 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:05:08.208160    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:05:19.744413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:05:19.744380    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:05:19.759532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:05:19.759507    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:05:20.480504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:05:20.480470    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 13:05:38.210798 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:05:38.210708    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:06:07.665714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.665681    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"]
Oct 30 13:06:07.668393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.668377    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.698424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.698397    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Oct 30 13:06:07.698424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.698397    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Oct 30 13:06:07.698595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.698448    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-b2hzx\""
Oct 30 13:06:07.699586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.699561    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"]
Oct 30 13:06:07.770538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.770503    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.770713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.770561    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.770766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.770710    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlr6b\" (UniqueName: \"kubernetes.io/projected/a65d7fdf-9a99-491d-8ffd-d075a47e123a-kube-api-access-rlr6b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.871784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.871746    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.871962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.871799    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlr6b\" (UniqueName: \"kubernetes.io/projected/a65d7fdf-9a99-491d-8ffd-d075a47e123a-kube-api-access-rlr6b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.871962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.871848    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.872223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.872203    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.872258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.872211    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.898994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.898960    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlr6b\" (UniqueName: \"kubernetes.io/projected/a65d7fdf-9a99-491d-8ffd-d075a47e123a-kube-api-access-rlr6b\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:07.981111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:07.981024    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:08.209152 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:06:08.209113    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:06:08.280955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.280886    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"]
Oct 30 13:06:08.284097 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:06:08.284069    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda65d7fdf_9a99_491d_8ffd_d075a47e123a.slice/crio-14727bd0d0d4e694f58f49d022eb1fe11d61ea40c26ee91bacc1d2ebb0863e6d WatchSource:0}: Error finding container 14727bd0d0d4e694f58f49d022eb1fe11d61ea40c26ee91bacc1d2ebb0863e6d: Status 404 returned error can't find the container with id 14727bd0d0d4e694f58f49d022eb1fe11d61ea40c26ee91bacc1d2ebb0863e6d
Oct 30 13:06:08.435546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.435505    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"]
Oct 30 13:06:08.438272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.438247    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.453437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.453403    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"openshift-ai-inference-openshift-ai-inference-dockercfg-slmf2\""
Oct 30 13:06:08.458130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.458100    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 13:06:08.466103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.466076    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 13:06:08.466103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.466076    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 13:06:08.467213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.467184    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"]
Oct 30 13:06:08.476509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476485    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.476678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476537    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.476678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476626    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029daaee-d17d-4310-b659-df585ef13a0e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.476817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476699    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px4c8\" (UniqueName: \"kubernetes.io/projected/029daaee-d17d-4310-b659-df585ef13a0e-kube-api-access-px4c8\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.476817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476761    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029daaee-d17d-4310-b659-df585ef13a0e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.476817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476793    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029daaee-d17d-4310-b659-df585ef13a0e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.476817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476816    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.477122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476842    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.477122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.476948    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.577670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577547    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029daaee-d17d-4310-b659-df585ef13a0e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.577670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577587    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px4c8\" (UniqueName: \"kubernetes.io/projected/029daaee-d17d-4310-b659-df585ef13a0e-kube-api-access-px4c8\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.577670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577607    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029daaee-d17d-4310-b659-df585ef13a0e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.577670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577627    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029daaee-d17d-4310-b659-df585ef13a0e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.577670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577671    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577691    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577723    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577751    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.577811    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.578141    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-workload-certs\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.578207    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-credential-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.578260    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-istio-data\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.578319    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-workload-socket\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.578470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.578449    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/029daaee-d17d-4310-b659-df585ef13a0e-istiod-ca-cert\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.580006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.579987    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/029daaee-d17d-4310-b659-df585ef13a0e-istio-envoy\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.580251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.580233    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/029daaee-d17d-4310-b659-df585ef13a0e-istio-podinfo\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.597487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.597468    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/029daaee-d17d-4310-b659-df585ef13a0e-istio-token\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.597680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.597660    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px4c8\" (UniqueName: \"kubernetes.io/projected/029daaee-d17d-4310-b659-df585ef13a0e-kube-api-access-px4c8\") pod \"openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595\" (UID: \"029daaee-d17d-4310-b659-df585ef13a0e\") " pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:08.629137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.629105    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2" event={"ID":"a65d7fdf-9a99-491d-8ffd-d075a47e123a","Type":"ContainerStarted","Data":"14727bd0d0d4e694f58f49d022eb1fe11d61ea40c26ee91bacc1d2ebb0863e6d"}
Oct 30 13:06:08.755657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:08.755596    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:09.079629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:09.079596    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"]
Oct 30 13:06:09.086311 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:06:09.086283    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod029daaee_d17d_4310_b659_df585ef13a0e.slice/crio-27f00a6624f60f90583c87334a7ba8393d785d51d2dcd65518cf44eec1860217 WatchSource:0}: Error finding container 27f00a6624f60f90583c87334a7ba8393d785d51d2dcd65518cf44eec1860217: Status 404 returned error can't find the container with id 27f00a6624f60f90583c87334a7ba8393d785d51d2dcd65518cf44eec1860217
Oct 30 13:06:09.636951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:09.636913    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595" event={"ID":"029daaee-d17d-4310-b659-df585ef13a0e","Type":"ContainerStarted","Data":"27f00a6624f60f90583c87334a7ba8393d785d51d2dcd65518cf44eec1860217"}
Oct 30 13:06:11.846175 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:11.846140    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:06:11.846589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:11.846224    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:06:11.846589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:11.846257    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:06:12.652146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:12.652108    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595" event={"ID":"029daaee-d17d-4310-b659-df585ef13a0e","Type":"ContainerStarted","Data":"f2ffc07ad26983ab07ec0221515a1fdfac23fec0eb6738a7b21eb23ad41e82b8"}
Oct 30 13:06:12.708249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:12.708148    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595" podStartSLOduration=1.952903923 podStartE2EDuration="4.708128482s" totalImagesPullingTime="2.755224559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:12.707498826 +0000 UTC m=+182.967737059" watchObservedRunningTime="2026-10-30 13:06:12.708128482 +0000 UTC m=+182.968366717"
Oct 30 13:06:12.756770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:12.756735    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:12.762056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:12.762030    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:13.656070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:13.656038    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:13.656975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:13.656954    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595"
Oct 30 13:06:14.664735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:14.664697    4291 generic.go:356] "Generic (PLEG): container finished" podID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerID="b3b264834a75b9e4cbe44964a08d88ccc9d7b2fc1fb0e2308909400202c10fd8" exitCode=0
Oct 30 13:06:14.665245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:14.664779    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2" event={"ID":"a65d7fdf-9a99-491d-8ffd-d075a47e123a","Type":"ContainerDied","Data":"b3b264834a75b9e4cbe44964a08d88ccc9d7b2fc1fb0e2308909400202c10fd8"}
Oct 30 13:06:15.670028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:15.669994    4291 generic.go:356] "Generic (PLEG): container finished" podID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerID="1c9a1fee0daf50824b8fd9c141bb55c440fdbb5ee3af921f22b4df1543b4a089" exitCode=0
Oct 30 13:06:15.670484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:15.670073    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2" event={"ID":"a65d7fdf-9a99-491d-8ffd-d075a47e123a","Type":"ContainerDied","Data":"1c9a1fee0daf50824b8fd9c141bb55c440fdbb5ee3af921f22b4df1543b4a089"}
Oct 30 13:06:21.695402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:21.695370    4291 generic.go:356] "Generic (PLEG): container finished" podID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerID="fdfd176b022eddd2f9db1aa6183064ee597e65a230189413718de1e54d9b52aa" exitCode=0
Oct 30 13:06:21.695799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:21.695447    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2" event={"ID":"a65d7fdf-9a99-491d-8ffd-d075a47e123a","Type":"ContainerDied","Data":"fdfd176b022eddd2f9db1aa6183064ee597e65a230189413718de1e54d9b52aa"}
Oct 30 13:06:22.822315 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:22.822290    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:22.911416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:22.911380    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-util\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-util\") pod \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") "
Oct 30 13:06:22.911588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:22.911456    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-bundle\") pod \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") "
Oct 30 13:06:22.911588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:22.911564    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a65d7fdf-9a99-491d-8ffd-d075a47e123a-kube-api-access-rlr6b\" (UniqueName: \"kubernetes.io/projected/a65d7fdf-9a99-491d-8ffd-d075a47e123a-kube-api-access-rlr6b\") pod \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\" (UID: \"a65d7fdf-9a99-491d-8ffd-d075a47e123a\") "
Oct 30 13:06:22.911963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:22.911932    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-bundle" pod "a65d7fdf-9a99-491d-8ffd-d075a47e123a" (UID: "a65d7fdf-9a99-491d-8ffd-d075a47e123a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:22.913737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:22.913709    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a65d7fdf-9a99-491d-8ffd-d075a47e123a-kube-api-access-rlr6b" pod "a65d7fdf-9a99-491d-8ffd-d075a47e123a" (UID: "a65d7fdf-9a99-491d-8ffd-d075a47e123a"). InnerVolumeSpecName "kube-api-access-rlr6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:06:22.915751 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:22.915730    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-util" pod "a65d7fdf-9a99-491d-8ffd-d075a47e123a" (UID: "a65d7fdf-9a99-491d-8ffd-d075a47e123a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:06:23.013028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:23.012951    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlr6b\" (UniqueName: \"kubernetes.io/projected/a65d7fdf-9a99-491d-8ffd-d075a47e123a-kube-api-access-rlr6b\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:06:23.013028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:23.012980    4291 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-util\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:06:23.013028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:23.012990    4291 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a65d7fdf-9a99-491d-8ffd-d075a47e123a-bundle\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:06:23.704943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:23.704913    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2" event={"ID":"a65d7fdf-9a99-491d-8ffd-d075a47e123a","Type":"ContainerDied","Data":"14727bd0d0d4e694f58f49d022eb1fe11d61ea40c26ee91bacc1d2ebb0863e6d"}
Oct 30 13:06:23.705115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:23.704962    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14727bd0d0d4e694f58f49d022eb1fe11d61ea40c26ee91bacc1d2ebb0863e6d"
Oct 30 13:06:23.705115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:23.704920    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cv68n2"
Oct 30 13:06:37.316652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.316605    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"]
Oct 30 13:06:37.317236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.316912    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerName="util"
Oct 30 13:06:37.317236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.316927    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerName="util"
Oct 30 13:06:37.317236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.316953    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerName="pull"
Oct 30 13:06:37.317236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.316959    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerName="pull"
Oct 30 13:06:37.317236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.316966    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerName="extract"
Oct 30 13:06:37.317236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.316971    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerName="extract"
Oct 30 13:06:37.317236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.317028    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="a65d7fdf-9a99-491d-8ffd-d075a47e123a" containerName="extract"
Oct 30 13:06:37.374523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.374488    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"]
Oct 30 13:06:37.374700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.374613    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.399196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.399166    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 13:06:37.399679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.399629    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 13:06:37.399805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.399704    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-gfqwn\""
Oct 30 13:06:37.399805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.399702    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 13:06:37.409681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.409659    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 13:06:37.409681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.409661    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 13:06:37.535786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.535741    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.535959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.535797    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/422ececd-13f2-4d1f-8bec-8e6839b2855b-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.535959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.535919    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skwsm\" (UniqueName: \"kubernetes.io/projected/422ececd-13f2-4d1f-8bec-8e6839b2855b-kube-api-access-skwsm\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.536033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.535966    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.536033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.535988    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.536033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.536009    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/422ececd-13f2-4d1f-8bec-8e6839b2855b-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.637373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.637329    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skwsm\" (UniqueName: \"kubernetes.io/projected/422ececd-13f2-4d1f-8bec-8e6839b2855b-kube-api-access-skwsm\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.637574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.637383    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.637574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.637411    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.637574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.637443    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/422ececd-13f2-4d1f-8bec-8e6839b2855b-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.637574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.637506    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.637574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.637545    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/422ececd-13f2-4d1f-8bec-8e6839b2855b-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.638056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.638025    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/422ececd-13f2-4d1f-8bec-8e6839b2855b-manifests\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.639873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.639851    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/422ececd-13f2-4d1f-8bec-8e6839b2855b-tmp\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.640155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.640137    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-webhook-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.640235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.640213    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-metrics-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.640270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.640229    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/422ececd-13f2-4d1f-8bec-8e6839b2855b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.657176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.657148    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skwsm\" (UniqueName: \"kubernetes.io/projected/422ececd-13f2-4d1f-8bec-8e6839b2855b-kube-api-access-skwsm\") pod \"opendatahub-operator-controller-manager-6dbbdb4cd4-blczh\" (UID: \"422ececd-13f2-4d1f-8bec-8e6839b2855b\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:37.711982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:37.711944    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:38.073224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:38.073143    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"]
Oct 30 13:06:38.075533 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:06:38.075508    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod422ececd_13f2_4d1f_8bec_8e6839b2855b.slice/crio-c6e249d3b84d0a2adc4703c74f7528c5bbad5773ce890e94404f51b6274b984d WatchSource:0}: Error finding container c6e249d3b84d0a2adc4703c74f7528c5bbad5773ce890e94404f51b6274b984d: Status 404 returned error can't find the container with id c6e249d3b84d0a2adc4703c74f7528c5bbad5773ce890e94404f51b6274b984d
Oct 30 13:06:38.209964 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:06:38.209922    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:06:38.756970 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:38.756925    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh" event={"ID":"422ececd-13f2-4d1f-8bec-8e6839b2855b","Type":"ContainerStarted","Data":"c6e249d3b84d0a2adc4703c74f7528c5bbad5773ce890e94404f51b6274b984d"}
Oct 30 13:06:41.772091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:41.772054    4291 generic.go:356] "Generic (PLEG): container finished" podID="422ececd-13f2-4d1f-8bec-8e6839b2855b" containerID="9cf1d137dbac0a646b9dab1e868335569e2392ded1adea33a8b7e856fad7ccd2" exitCode=0
Oct 30 13:06:41.772490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:41.772126    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh" event={"ID":"422ececd-13f2-4d1f-8bec-8e6839b2855b","Type":"ContainerDied","Data":"9cf1d137dbac0a646b9dab1e868335569e2392ded1adea33a8b7e856fad7ccd2"}
Oct 30 13:06:42.778346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:42.778308    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh" event={"ID":"422ececd-13f2-4d1f-8bec-8e6839b2855b","Type":"ContainerStarted","Data":"8a10c1a17bd5b94f57ad12fc28c389b47db22b1325f3a5fdb6c2aff89f79f995"}
Oct 30 13:06:42.778910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:42.778429    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:42.873949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:42.873881    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh" podStartSLOduration=2.421306787 podStartE2EDuration="5.873856069s" totalImagesPullingTime="3.036581418s" totalInitContainerRuntime="415.967864ms" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:37 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:42.86030719 +0000 UTC m=+213.120545423" watchObservedRunningTime="2026-10-30 13:06:42.873856069 +0000 UTC m=+213.134094304"
Oct 30 13:06:51.045661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.045614    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"]
Oct 30 13:06:51.049838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.049821    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.077263 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.077240    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 13:06:51.077263 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.077239    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 13:06:51.077439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.077239    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-7gp2k\""
Oct 30 13:06:51.078527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.078510    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"]
Oct 30 13:06:51.144224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.144189    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e8db0-f508-4559-8f49-f929a5745a7a-tmp\") pod \"jobset-operator-674cd59dc5-twn5s\" (UID: \"e75e8db0-f508-4559-8f49-f929a5745a7a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.144385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.144229    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7wmq\" (UniqueName: \"kubernetes.io/projected/e75e8db0-f508-4559-8f49-f929a5745a7a-kube-api-access-b7wmq\") pod \"jobset-operator-674cd59dc5-twn5s\" (UID: \"e75e8db0-f508-4559-8f49-f929a5745a7a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.245328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.245288    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e8db0-f508-4559-8f49-f929a5745a7a-tmp\") pod \"jobset-operator-674cd59dc5-twn5s\" (UID: \"e75e8db0-f508-4559-8f49-f929a5745a7a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.245487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.245333    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7wmq\" (UniqueName: \"kubernetes.io/projected/e75e8db0-f508-4559-8f49-f929a5745a7a-kube-api-access-b7wmq\") pod \"jobset-operator-674cd59dc5-twn5s\" (UID: \"e75e8db0-f508-4559-8f49-f929a5745a7a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.245722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.245704    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e75e8db0-f508-4559-8f49-f929a5745a7a-tmp\") pod \"jobset-operator-674cd59dc5-twn5s\" (UID: \"e75e8db0-f508-4559-8f49-f929a5745a7a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.265070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.265047    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7wmq\" (UniqueName: \"kubernetes.io/projected/e75e8db0-f508-4559-8f49-f929a5745a7a-kube-api-access-b7wmq\") pod \"jobset-operator-674cd59dc5-twn5s\" (UID: \"e75e8db0-f508-4559-8f49-f929a5745a7a\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.369961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.369925    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"
Oct 30 13:06:51.372962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.372941    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"]
Oct 30 13:06:51.375100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.375085    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.392078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.392040    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 13:06:51.392297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.392281    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 13:06:51.394499 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.394465    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"]
Oct 30 13:06:51.433407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.433356    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-nggwq\""
Oct 30 13:06:51.547794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.547763    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1758afc-5666-47ab-9a6e-9158f962d5b7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tt6tr\" (UID: \"b1758afc-5666-47ab-9a6e-9158f962d5b7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.547963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.547810    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g92vz\" (UniqueName: \"kubernetes.io/projected/b1758afc-5666-47ab-9a6e-9158f962d5b7-kube-api-access-g92vz\") pod \"cert-manager-cainjector-5c57bbd4fc-tt6tr\" (UID: \"b1758afc-5666-47ab-9a6e-9158f962d5b7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.648655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.648569    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g92vz\" (UniqueName: \"kubernetes.io/projected/b1758afc-5666-47ab-9a6e-9158f962d5b7-kube-api-access-g92vz\") pod \"cert-manager-cainjector-5c57bbd4fc-tt6tr\" (UID: \"b1758afc-5666-47ab-9a6e-9158f962d5b7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.648798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.648667    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1758afc-5666-47ab-9a6e-9158f962d5b7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tt6tr\" (UID: \"b1758afc-5666-47ab-9a6e-9158f962d5b7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.668375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.668344    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s"]
Oct 30 13:06:51.670834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.670813    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b1758afc-5666-47ab-9a6e-9158f962d5b7-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-tt6tr\" (UID: \"b1758afc-5666-47ab-9a6e-9158f962d5b7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.670937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.670855    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g92vz\" (UniqueName: \"kubernetes.io/projected/b1758afc-5666-47ab-9a6e-9158f962d5b7-kube-api-access-g92vz\") pod \"cert-manager-cainjector-5c57bbd4fc-tt6tr\" (UID: \"b1758afc-5666-47ab-9a6e-9158f962d5b7\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.672122 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:06:51.672096    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode75e8db0_f508_4559_8f49_f929a5745a7a.slice/crio-708a5ffc61f41f1b93bf5dc24fa24422cf5c74576160fe0e967a2deaf9371c93 WatchSource:0}: Error finding container 708a5ffc61f41f1b93bf5dc24fa24422cf5c74576160fe0e967a2deaf9371c93: Status 404 returned error can't find the container with id 708a5ffc61f41f1b93bf5dc24fa24422cf5c74576160fe0e967a2deaf9371c93
Oct 30 13:06:51.699268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.699230    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"
Oct 30 13:06:51.810520 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.810484    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s" event={"ID":"e75e8db0-f508-4559-8f49-f929a5745a7a","Type":"ContainerStarted","Data":"708a5ffc61f41f1b93bf5dc24fa24422cf5c74576160fe0e967a2deaf9371c93"}
Oct 30 13:06:51.996212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:51.996187    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr"]
Oct 30 13:06:51.998560 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:06:51.998528    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1758afc_5666_47ab_9a6e_9158f962d5b7.slice/crio-f718b8db427bac4a13fb97196376bdd70ebf57810bea276021d69116f731c5bd WatchSource:0}: Error finding container f718b8db427bac4a13fb97196376bdd70ebf57810bea276021d69116f731c5bd: Status 404 returned error can't find the container with id f718b8db427bac4a13fb97196376bdd70ebf57810bea276021d69116f731c5bd
Oct 30 13:06:52.814772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:52.814697    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr" event={"ID":"b1758afc-5666-47ab-9a6e-9158f962d5b7","Type":"ContainerStarted","Data":"f718b8db427bac4a13fb97196376bdd70ebf57810bea276021d69116f731c5bd"}
Oct 30 13:06:53.784747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:53.784714    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-6dbbdb4cd4-blczh"
Oct 30 13:06:55.833999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:55.833959    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s" event={"ID":"e75e8db0-f508-4559-8f49-f929a5745a7a","Type":"ContainerStarted","Data":"8f56338cf8776a000fc39e12cf7564dfdf4e73d64e1e57cd099b1df94782a86f"}
Oct 30 13:06:55.874261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:55.874199    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-twn5s" podStartSLOduration=1.2270668869999999 podStartE2EDuration="4.874179603s" totalImagesPullingTime="3.647112716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:55.873419211 +0000 UTC m=+226.133657446" watchObservedRunningTime="2026-10-30 13:06:55.874179603 +0000 UTC m=+226.134417838"
Oct 30 13:06:56.839091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:56.839047    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr" event={"ID":"b1758afc-5666-47ab-9a6e-9158f962d5b7","Type":"ContainerStarted","Data":"2bbe58e52f732ddd6429753de39f8032b3b729932b39f907aa4fc7a7466c9d13"}
Oct 30 13:06:56.887083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:06:56.887032    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-tt6tr" podStartSLOduration=1.9135556889999998 podStartE2EDuration="5.887017611s" totalImagesPullingTime="3.973461922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:06:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:06:56.88583858 +0000 UTC m=+227.146076813" watchObservedRunningTime="2026-10-30 13:06:56.887017611 +0000 UTC m=+227.147255845"
Oct 30 13:07:00.278517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.278483    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-nq5bx"]
Oct 30 13:07:00.280745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.280727    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.309217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.309194    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-gznrz\""
Oct 30 13:07:00.310476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.310456    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-nq5bx"]
Oct 30 13:07:00.327746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.327722    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dqv4\" (UniqueName: \"kubernetes.io/projected/b043d8e4-5d70-46bd-9e2f-a590d6dea977-kube-api-access-2dqv4\") pod \"cert-manager-597b95cb84-nq5bx\" (UID: \"b043d8e4-5d70-46bd-9e2f-a590d6dea977\") " pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.327886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.327784    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b043d8e4-5d70-46bd-9e2f-a590d6dea977-bound-sa-token\") pod \"cert-manager-597b95cb84-nq5bx\" (UID: \"b043d8e4-5d70-46bd-9e2f-a590d6dea977\") " pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.429192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.429155    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dqv4\" (UniqueName: \"kubernetes.io/projected/b043d8e4-5d70-46bd-9e2f-a590d6dea977-kube-api-access-2dqv4\") pod \"cert-manager-597b95cb84-nq5bx\" (UID: \"b043d8e4-5d70-46bd-9e2f-a590d6dea977\") " pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.429388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.429223    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b043d8e4-5d70-46bd-9e2f-a590d6dea977-bound-sa-token\") pod \"cert-manager-597b95cb84-nq5bx\" (UID: \"b043d8e4-5d70-46bd-9e2f-a590d6dea977\") " pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.448705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.448681    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b043d8e4-5d70-46bd-9e2f-a590d6dea977-bound-sa-token\") pod \"cert-manager-597b95cb84-nq5bx\" (UID: \"b043d8e4-5d70-46bd-9e2f-a590d6dea977\") " pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.448811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.448772    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dqv4\" (UniqueName: \"kubernetes.io/projected/b043d8e4-5d70-46bd-9e2f-a590d6dea977-kube-api-access-2dqv4\") pod \"cert-manager-597b95cb84-nq5bx\" (UID: \"b043d8e4-5d70-46bd-9e2f-a590d6dea977\") " pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.590367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.590283    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-nq5bx"
Oct 30 13:07:00.884597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:00.884563    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-nq5bx"]
Oct 30 13:07:00.888055 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:07:00.888029    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb043d8e4_5d70_46bd_9e2f_a590d6dea977.slice/crio-9e007eb82b7b0a00fb9061c5ddf7031727d78f261a01694fab10fd7f53c7be47 WatchSource:0}: Error finding container 9e007eb82b7b0a00fb9061c5ddf7031727d78f261a01694fab10fd7f53c7be47: Status 404 returned error can't find the container with id 9e007eb82b7b0a00fb9061c5ddf7031727d78f261a01694fab10fd7f53c7be47
Oct 30 13:07:01.860432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:01.860396    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-nq5bx" event={"ID":"b043d8e4-5d70-46bd-9e2f-a590d6dea977","Type":"ContainerStarted","Data":"8652e2bfdf1f6d62b6a13ce5bf600db75eb06d70450e01401046ac3abc271b72"}
Oct 30 13:07:01.860432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:01.860439    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-nq5bx" event={"ID":"b043d8e4-5d70-46bd-9e2f-a590d6dea977","Type":"ContainerStarted","Data":"9e007eb82b7b0a00fb9061c5ddf7031727d78f261a01694fab10fd7f53c7be47"}
Oct 30 13:07:01.911115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:01.911069    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-nq5bx" podStartSLOduration=1.91105451 podStartE2EDuration="1.91105451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:01.909700213 +0000 UTC m=+232.169938447" watchObservedRunningTime="2026-10-30 13:07:01.91105451 +0000 UTC m=+232.171292744"
Oct 30 13:07:08.208935 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:07:08.208901    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:07:16.073235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.073197    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"]
Oct 30 13:07:16.081083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.081060    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.094952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.094925    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"]
Oct 30 13:07:16.096417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.096398    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 13:07:16.096540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.096503    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 13:07:16.096540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.096504    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 13:07:16.096665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.096649    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 13:07:16.096870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.096857    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-x4w5v\""
Oct 30 13:07:16.153019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.152979    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-527xf\" (UniqueName: \"kubernetes.io/projected/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-kube-api-access-527xf\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.153212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.153043    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.153212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.153131    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.253993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.253941    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.253993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.253998    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.254209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.254044    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-527xf\" (UniqueName: \"kubernetes.io/projected/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-kube-api-access-527xf\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.254587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.254566    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.256470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.256451    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.273248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.273220    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-527xf\" (UniqueName: \"kubernetes.io/projected/1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c-kube-api-access-527xf\") pod \"kuadrant-console-plugin-66c4fb575d-qcjns\" (UID: \"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.391858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.391819    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"
Oct 30 13:07:16.692456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.692302    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns"]
Oct 30 13:07:16.694953 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:07:16.694924    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c5f67ee_d4e7_4781_908e_2e2e6e0fc88c.slice/crio-f87ee900984d413d6f969dc80ca4377eacee8a009526c727bf7f5ed8306f5063 WatchSource:0}: Error finding container f87ee900984d413d6f969dc80ca4377eacee8a009526c727bf7f5ed8306f5063: Status 404 returned error can't find the container with id f87ee900984d413d6f969dc80ca4377eacee8a009526c727bf7f5ed8306f5063
Oct 30 13:07:16.916241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:16.916206    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns" event={"ID":"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c","Type":"ContainerStarted","Data":"f87ee900984d413d6f969dc80ca4377eacee8a009526c727bf7f5ed8306f5063"}
Oct 30 13:07:17.164823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.164785    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7998d67549-78bfj"]
Oct 30 13:07:17.168455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.168433    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.201114 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.201063    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7998d67549-78bfj"]
Oct 30 13:07:17.264663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.264594    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631405e4-1521-4f58-9a86-9d1e55e60e73-tls-certs\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.264836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.264731    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c79j7\" (UniqueName: \"kubernetes.io/projected/631405e4-1521-4f58-9a86-9d1e55e60e73-kube-api-access-c79j7\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.264836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.264799    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631405e4-1521-4f58-9a86-9d1e55e60e73-tmp\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.275576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.275545    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 13:07:17.275746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.275719    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-5nkmk\""
Oct 30 13:07:17.275805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.275763    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 13:07:17.365879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.365845    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c79j7\" (UniqueName: \"kubernetes.io/projected/631405e4-1521-4f58-9a86-9d1e55e60e73-kube-api-access-c79j7\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.366052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.365913    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631405e4-1521-4f58-9a86-9d1e55e60e73-tmp\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.366052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.365964    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631405e4-1521-4f58-9a86-9d1e55e60e73-tls-certs\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.369125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.369064    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/631405e4-1521-4f58-9a86-9d1e55e60e73-tls-certs\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.370172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.370125    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/631405e4-1521-4f58-9a86-9d1e55e60e73-tmp\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.392430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.392396    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c79j7\" (UniqueName: \"kubernetes.io/projected/631405e4-1521-4f58-9a86-9d1e55e60e73-kube-api-access-c79j7\") pod \"kube-auth-proxy-7998d67549-78bfj\" (UID: \"631405e4-1521-4f58-9a86-9d1e55e60e73\") " pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.480136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.479828    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj"
Oct 30 13:07:17.805905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.805827    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7998d67549-78bfj"]
Oct 30 13:07:17.814664 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:07:17.814614    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod631405e4_1521_4f58_9a86_9d1e55e60e73.slice/crio-53471f4b2a0b0cc7a4b8aee648400802b1af7817adb9a831f913a690a9cd5fe1 WatchSource:0}: Error finding container 53471f4b2a0b0cc7a4b8aee648400802b1af7817adb9a831f913a690a9cd5fe1: Status 404 returned error can't find the container with id 53471f4b2a0b0cc7a4b8aee648400802b1af7817adb9a831f913a690a9cd5fe1
Oct 30 13:07:17.927226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:17.927137    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj" event={"ID":"631405e4-1521-4f58-9a86-9d1e55e60e73","Type":"ContainerStarted","Data":"53471f4b2a0b0cc7a4b8aee648400802b1af7817adb9a831f913a690a9cd5fe1"}
Oct 30 13:07:20.427221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.425951    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:07:20.433914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.433884    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jj6cl"]
Oct 30 13:07:20.434065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.434034    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.436982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.436961    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl"
Oct 30 13:07:20.448503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.448477    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jj6cl"]
Oct 30 13:07:20.450223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.450205    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:07:20.458455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.458429    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 13:07:20.459478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.459453    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sxnrz\""
Oct 30 13:07:20.495400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.495365    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65md9\" (UniqueName: \"kubernetes.io/projected/6ceae9cc-460e-472f-a47a-e78fa8b187a1-kube-api-access-65md9\") pod \"authorino-56c6c9ddd5-jj6cl\" (UID: \"6ceae9cc-460e-472f-a47a-e78fa8b187a1\") " pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl"
Oct 30 13:07:20.495572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.495493    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46a07241-a5d6-4177-899b-a851d4760824-config-file\") pod \"limitador-limitador-7c9b87d574-xfsxp\" (UID: \"46a07241-a5d6-4177-899b-a851d4760824\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.495572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.495544    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp84g\" (UniqueName: \"kubernetes.io/projected/46a07241-a5d6-4177-899b-a851d4760824-kube-api-access-bp84g\") pod \"limitador-limitador-7c9b87d574-xfsxp\" (UID: \"46a07241-a5d6-4177-899b-a851d4760824\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.515106 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.515075    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:07:20.596268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.596240    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp84g\" (UniqueName: \"kubernetes.io/projected/46a07241-a5d6-4177-899b-a851d4760824-kube-api-access-bp84g\") pod \"limitador-limitador-7c9b87d574-xfsxp\" (UID: \"46a07241-a5d6-4177-899b-a851d4760824\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.596400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.596280    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65md9\" (UniqueName: \"kubernetes.io/projected/6ceae9cc-460e-472f-a47a-e78fa8b187a1-kube-api-access-65md9\") pod \"authorino-56c6c9ddd5-jj6cl\" (UID: \"6ceae9cc-460e-472f-a47a-e78fa8b187a1\") " pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl"
Oct 30 13:07:20.596400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.596380    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46a07241-a5d6-4177-899b-a851d4760824-config-file\") pod \"limitador-limitador-7c9b87d574-xfsxp\" (UID: \"46a07241-a5d6-4177-899b-a851d4760824\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.597107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.597074    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/46a07241-a5d6-4177-899b-a851d4760824-config-file\") pod \"limitador-limitador-7c9b87d574-xfsxp\" (UID: \"46a07241-a5d6-4177-899b-a851d4760824\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.623086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.622986    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65md9\" (UniqueName: \"kubernetes.io/projected/6ceae9cc-460e-472f-a47a-e78fa8b187a1-kube-api-access-65md9\") pod \"authorino-56c6c9ddd5-jj6cl\" (UID: \"6ceae9cc-460e-472f-a47a-e78fa8b187a1\") " pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl"
Oct 30 13:07:20.623392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.623374    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp84g\" (UniqueName: \"kubernetes.io/projected/46a07241-a5d6-4177-899b-a851d4760824-kube-api-access-bp84g\") pod \"limitador-limitador-7c9b87d574-xfsxp\" (UID: \"46a07241-a5d6-4177-899b-a851d4760824\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.756867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.756786    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:20.764383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.764358    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl"
Oct 30 13:07:20.945728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.945681    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj" event={"ID":"631405e4-1521-4f58-9a86-9d1e55e60e73","Type":"ContainerStarted","Data":"a16ea7496441d5fe99d9a899e6722035b6399f461e15f0b51f5301d056d06383"}
Oct 30 13:07:20.987865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:20.987808    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7998d67549-78bfj" podStartSLOduration=1.222026449 podStartE2EDuration="3.987789962s" totalImagesPullingTime="2.765763513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:20.985516385 +0000 UTC m=+251.245754619" watchObservedRunningTime="2026-10-30 13:07:20.987789962 +0000 UTC m=+251.248028198"
Oct 30 13:07:21.064449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:21.064381    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:07:21.067802 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:07:21.067773    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46a07241_a5d6_4177_899b_a851d4760824.slice/crio-38f9a733096dd7baee327a44e0d90ef4e9cfcda60b4985203fbe2f782f444440 WatchSource:0}: Error finding container 38f9a733096dd7baee327a44e0d90ef4e9cfcda60b4985203fbe2f782f444440: Status 404 returned error can't find the container with id 38f9a733096dd7baee327a44e0d90ef4e9cfcda60b4985203fbe2f782f444440
Oct 30 13:07:21.077404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:21.077376    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jj6cl"]
Oct 30 13:07:21.081789 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:07:21.081757    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ceae9cc_460e_472f_a47a_e78fa8b187a1.slice/crio-6d688b35c715ccee61527b92542326e34b7e1572f2d6cea3c8ea88faafe5df9c WatchSource:0}: Error finding container 6d688b35c715ccee61527b92542326e34b7e1572f2d6cea3c8ea88faafe5df9c: Status 404 returned error can't find the container with id 6d688b35c715ccee61527b92542326e34b7e1572f2d6cea3c8ea88faafe5df9c
Oct 30 13:07:21.952827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:21.952756    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl" event={"ID":"6ceae9cc-460e-472f-a47a-e78fa8b187a1","Type":"ContainerStarted","Data":"6d688b35c715ccee61527b92542326e34b7e1572f2d6cea3c8ea88faafe5df9c"}
Oct 30 13:07:21.955458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:21.955417    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp" event={"ID":"46a07241-a5d6-4177-899b-a851d4760824","Type":"ContainerStarted","Data":"38f9a733096dd7baee327a44e0d90ef4e9cfcda60b4985203fbe2f782f444440"}
Oct 30 13:07:22.962113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:22.962017    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns" event={"ID":"1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c","Type":"ContainerStarted","Data":"34274d8c40f0f101b8aa267e9bb378a6dea11610171263ecdda06af07f434737"}
Oct 30 13:07:23.006156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:23.006098    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-qcjns" podStartSLOduration=1.092060957 podStartE2EDuration="7.006079946s" totalImagesPullingTime="5.914018989s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:23.005362263 +0000 UTC m=+253.265600510" watchObservedRunningTime="2026-10-30 13:07:23.006079946 +0000 UTC m=+253.266318179"
Oct 30 13:07:23.971274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:23.971235    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp" event={"ID":"46a07241-a5d6-4177-899b-a851d4760824","Type":"ContainerStarted","Data":"6a9b7b4c75383f7d41ab97bd0fc79decb3939b290fc3c003986cb7c68e3b1608"}
Oct 30 13:07:23.971759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:23.971471    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:24.018946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:24.018883    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp" podStartSLOduration=1.302177452 podStartE2EDuration="4.018866469s" totalImagesPullingTime="2.716689017s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:24.018132213 +0000 UTC m=+254.278370448" watchObservedRunningTime="2026-10-30 13:07:24.018866469 +0000 UTC m=+254.279104704"
Oct 30 13:07:24.976892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:24.976857    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl" event={"ID":"6ceae9cc-460e-472f-a47a-e78fa8b187a1","Type":"ContainerStarted","Data":"c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7"}
Oct 30 13:07:25.018695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:25.018616    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl" podStartSLOduration=1.963820524 podStartE2EDuration="5.018599168s" totalImagesPullingTime="3.054778644s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:07:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:07:25.016743737 +0000 UTC m=+255.276981971" watchObservedRunningTime="2026-10-30 13:07:25.018599168 +0000 UTC m=+255.278837402"
Oct 30 13:07:34.978214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:07:34.978183    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"
Oct 30 13:07:38.210470 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:07:38.210420    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:08:08.209496 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:08:08.209458    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:08:23.056515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.056437    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rlznc"]
Oct 30 13:08:23.060067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.059471    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.081010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.080985    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 13:08:23.081166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.081120    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rlznc"]
Oct 30 13:08:23.091329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.091303    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-vfnb6\""
Oct 30 13:08:23.091467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.091341    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 13:08:23.091467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.091372    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 13:08:23.108052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.108022    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"]
Oct 30 13:08:23.110358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.110342    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.130546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.130518    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"]
Oct 30 13:08:23.144182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.144146    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqdx4\" (UniqueName: \"kubernetes.io/projected/ff82d94e-deb9-496f-9078-348900979efe-kube-api-access-kqdx4\") pod \"kserve-controller-manager-776cc66d47-rlznc\" (UID: \"ff82d94e-deb9-496f-9078-348900979efe\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.144316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.144209    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff82d94e-deb9-496f-9078-348900979efe-cert\") pod \"kserve-controller-manager-776cc66d47-rlznc\" (UID: \"ff82d94e-deb9-496f-9078-348900979efe\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.173602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.173574    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-scfzp\""
Oct 30 13:08:23.174252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.174235    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 13:08:23.245081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.245048    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqdx4\" (UniqueName: \"kubernetes.io/projected/ff82d94e-deb9-496f-9078-348900979efe-kube-api-access-kqdx4\") pod \"kserve-controller-manager-776cc66d47-rlznc\" (UID: \"ff82d94e-deb9-496f-9078-348900979efe\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.245266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.245096    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff82d94e-deb9-496f-9078-348900979efe-cert\") pod \"kserve-controller-manager-776cc66d47-rlznc\" (UID: \"ff82d94e-deb9-496f-9078-348900979efe\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.245266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.245141    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfe531a-b76a-45a5-b344-4925b5ea4426-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p2vxr\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.245266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.245250    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bxfc\" (UniqueName: \"kubernetes.io/projected/9bfe531a-b76a-45a5-b344-4925b5ea4426-kube-api-access-4bxfc\") pod \"llmisvc-controller-manager-869d5dcc5d-p2vxr\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.247744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.247715    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ff82d94e-deb9-496f-9078-348900979efe-cert\") pod \"kserve-controller-manager-776cc66d47-rlznc\" (UID: \"ff82d94e-deb9-496f-9078-348900979efe\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.265478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.265451    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqdx4\" (UniqueName: \"kubernetes.io/projected/ff82d94e-deb9-496f-9078-348900979efe-kube-api-access-kqdx4\") pod \"kserve-controller-manager-776cc66d47-rlznc\" (UID: \"ff82d94e-deb9-496f-9078-348900979efe\") " pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.345663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.345603    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfe531a-b76a-45a5-b344-4925b5ea4426-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p2vxr\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.345836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.345672    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bxfc\" (UniqueName: \"kubernetes.io/projected/9bfe531a-b76a-45a5-b344-4925b5ea4426-kube-api-access-4bxfc\") pod \"llmisvc-controller-manager-869d5dcc5d-p2vxr\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.348000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.347977    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfe531a-b76a-45a5-b344-4925b5ea4426-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-p2vxr\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.365138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.365113    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bxfc\" (UniqueName: \"kubernetes.io/projected/9bfe531a-b76a-45a5-b344-4925b5ea4426-kube-api-access-4bxfc\") pod \"llmisvc-controller-manager-869d5dcc5d-p2vxr\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.373935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.373913    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:23.422868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.422830    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:23.673745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.673670    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-rlznc"]
Oct 30 13:08:23.676727 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:08:23.676690    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff82d94e_deb9_496f_9078_348900979efe.slice/crio-8d154c97a650b4a96dea262a01961ca13adceffd1c7180b95a0219bd3a8e2cdd WatchSource:0}: Error finding container 8d154c97a650b4a96dea262a01961ca13adceffd1c7180b95a0219bd3a8e2cdd: Status 404 returned error can't find the container with id 8d154c97a650b4a96dea262a01961ca13adceffd1c7180b95a0219bd3a8e2cdd
Oct 30 13:08:23.678097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.678079    4291 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:08:23.725278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:23.725249    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"]
Oct 30 13:08:23.728106 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:08:23.728085    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod9bfe531a_b76a_45a5_b344_4925b5ea4426.slice/crio-91702af6891bc39ce8977d78c2f5d3c0695ab182847ddd8e9807bed87eaed419 WatchSource:0}: Error finding container 91702af6891bc39ce8977d78c2f5d3c0695ab182847ddd8e9807bed87eaed419: Status 404 returned error can't find the container with id 91702af6891bc39ce8977d78c2f5d3c0695ab182847ddd8e9807bed87eaed419
Oct 30 13:08:24.225307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:24.225274    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr" event={"ID":"9bfe531a-b76a-45a5-b344-4925b5ea4426","Type":"ContainerStarted","Data":"91702af6891bc39ce8977d78c2f5d3c0695ab182847ddd8e9807bed87eaed419"}
Oct 30 13:08:24.226696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:24.226669    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc" event={"ID":"ff82d94e-deb9-496f-9078-348900979efe","Type":"ContainerStarted","Data":"8d154c97a650b4a96dea262a01961ca13adceffd1c7180b95a0219bd3a8e2cdd"}
Oct 30 13:08:26.237447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:26.237414    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr" event={"ID":"9bfe531a-b76a-45a5-b344-4925b5ea4426","Type":"ContainerStarted","Data":"8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f"}
Oct 30 13:08:26.237908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:26.237530    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:26.239030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:26.239008    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc" event={"ID":"ff82d94e-deb9-496f-9078-348900979efe","Type":"ContainerStarted","Data":"0a9e85fa5ec197d0ea5d98750d6fd9b08f3d3abecfcb9b0c7a153464a29ce537"}
Oct 30 13:08:26.239153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:26.239124    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:08:26.275661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:26.275584    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr" podStartSLOduration=1.324911956 podStartE2EDuration="3.275565662s" totalImagesPullingTime="1.950653706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:26.275268503 +0000 UTC m=+316.535506736" watchObservedRunningTime="2026-10-30 13:08:26.275565662 +0000 UTC m=+316.535803899"
Oct 30 13:08:26.313482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:26.313434    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc" podStartSLOduration=1.26466583 podStartE2EDuration="3.313419558s" totalImagesPullingTime="2.048753728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:26.312402756 +0000 UTC m=+316.572641137" watchObservedRunningTime="2026-10-30 13:08:26.313419558 +0000 UTC m=+316.573657791"
Oct 30 13:08:37.735197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:37.735157    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:08:38.214338 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:08:38.214301    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:08:45.707036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.707001    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5fd7868fd-cqttk"]
Oct 30 13:08:45.711138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.711120    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:45.725436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.725414    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fd7868fd-cqttk"]
Oct 30 13:08:45.756137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.756116    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 13:08:45.840383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.840351    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glmp7\" (UniqueName: \"kubernetes.io/projected/e226fc68-12d0-48ec-a0ed-a588f3d032f1-kube-api-access-glmp7\") pod \"authorino-5fd7868fd-cqttk\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") " pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:45.840529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.840464    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e226fc68-12d0-48ec-a0ed-a588f3d032f1-tls-cert\") pod \"authorino-5fd7868fd-cqttk\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") " pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:45.941613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.941585    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e226fc68-12d0-48ec-a0ed-a588f3d032f1-tls-cert\") pod \"authorino-5fd7868fd-cqttk\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") " pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:45.941772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.941649    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glmp7\" (UniqueName: \"kubernetes.io/projected/e226fc68-12d0-48ec-a0ed-a588f3d032f1-kube-api-access-glmp7\") pod \"authorino-5fd7868fd-cqttk\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") " pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:45.944032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.944011    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e226fc68-12d0-48ec-a0ed-a588f3d032f1-tls-cert\") pod \"authorino-5fd7868fd-cqttk\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") " pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:45.960609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:45.960558    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glmp7\" (UniqueName: \"kubernetes.io/projected/e226fc68-12d0-48ec-a0ed-a588f3d032f1-kube-api-access-glmp7\") pod \"authorino-5fd7868fd-cqttk\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") " pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:46.022016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:46.021982    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:08:46.314658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:46.314623    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5fd7868fd-cqttk"]
Oct 30 13:08:46.317121 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:08:46.317085    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode226fc68_12d0_48ec_a0ed_a588f3d032f1.slice/crio-06625ff8920d265d64e403b178d5891ef112c93e7439696f47531436819f9728 WatchSource:0}: Error finding container 06625ff8920d265d64e403b178d5891ef112c93e7439696f47531436819f9728: Status 404 returned error can't find the container with id 06625ff8920d265d64e403b178d5891ef112c93e7439696f47531436819f9728
Oct 30 13:08:46.329056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:46.329026    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd7868fd-cqttk" event={"ID":"e226fc68-12d0-48ec-a0ed-a588f3d032f1","Type":"ContainerStarted","Data":"06625ff8920d265d64e403b178d5891ef112c93e7439696f47531436819f9728"}
Oct 30 13:08:47.336005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.335970    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd7868fd-cqttk" event={"ID":"e226fc68-12d0-48ec-a0ed-a588f3d032f1","Type":"ContainerStarted","Data":"a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c"}
Oct 30 13:08:47.375185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.375135    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5fd7868fd-cqttk" podStartSLOduration=2.019083947 podStartE2EDuration="2.375120637s" totalImagesPullingTime="356.03669ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:08:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:08:47.373599977 +0000 UTC m=+337.633838212" watchObservedRunningTime="2026-10-30 13:08:47.375120637 +0000 UTC m=+337.635358870"
Oct 30 13:08:47.406350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.406312    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jj6cl"]
Oct 30 13:08:47.406560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.406536    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl" podUID="6ceae9cc-460e-472f-a47a-e78fa8b187a1" containerName="authorino" containerID="cri-o://c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7" gracePeriod=30
Oct 30 13:08:47.655015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.654993    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl"
Oct 30 13:08:47.756703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.756676    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ceae9cc-460e-472f-a47a-e78fa8b187a1-kube-api-access-65md9\" (UniqueName: \"kubernetes.io/projected/6ceae9cc-460e-472f-a47a-e78fa8b187a1-kube-api-access-65md9\") pod \"6ceae9cc-460e-472f-a47a-e78fa8b187a1\" (UID: \"6ceae9cc-460e-472f-a47a-e78fa8b187a1\") "
Oct 30 13:08:47.758850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.758821    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ceae9cc-460e-472f-a47a-e78fa8b187a1-kube-api-access-65md9" pod "6ceae9cc-460e-472f-a47a-e78fa8b187a1" (UID: "6ceae9cc-460e-472f-a47a-e78fa8b187a1"). InnerVolumeSpecName "kube-api-access-65md9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:08:47.858173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:47.858144    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65md9\" (UniqueName: \"kubernetes.io/projected/6ceae9cc-460e-472f-a47a-e78fa8b187a1-kube-api-access-65md9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:08:48.341549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.341516    4291 generic.go:356] "Generic (PLEG): container finished" podID="6ceae9cc-460e-472f-a47a-e78fa8b187a1" containerID="c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7" exitCode=0
Oct 30 13:08:48.342082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.341563    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl"
Oct 30 13:08:48.342082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.341600    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl" event={"ID":"6ceae9cc-460e-472f-a47a-e78fa8b187a1","Type":"ContainerDied","Data":"c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7"}
Oct 30 13:08:48.342082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.341670    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-jj6cl" event={"ID":"6ceae9cc-460e-472f-a47a-e78fa8b187a1","Type":"ContainerDied","Data":"6d688b35c715ccee61527b92542326e34b7e1572f2d6cea3c8ea88faafe5df9c"}
Oct 30 13:08:48.342082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.341701    4291 scope.go:122] "RemoveContainer" containerID="c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7"
Oct 30 13:08:48.349729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.349713    4291 scope.go:122] "RemoveContainer" containerID="c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7"
Oct 30 13:08:48.349982 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:08:48.349963    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7\": container with ID starting with c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7 not found: ID does not exist" containerID="c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7"
Oct 30 13:08:48.350020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.349990    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7"} err="failed to get container status \"c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7\": rpc error: code = NotFound desc = could not find container \"c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7\": container with ID starting with c3c9e9ea4ff5cfc83e8bb2a7639359907f0120565a209628140402a923e26fb7 not found: ID does not exist"
Oct 30 13:08:48.390954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.390872    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jj6cl"]
Oct 30 13:08:48.394724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:48.394706    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-jj6cl"]
Oct 30 13:08:49.876120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:49.876090    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ceae9cc-460e-472f-a47a-e78fa8b187a1" path="/var/lib/kubelet/pods/6ceae9cc-460e-472f-a47a-e78fa8b187a1/volumes"
Oct 30 13:08:57.245747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:57.245717    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:08:57.248663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:08:57.248628    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-rlznc"
Oct 30 13:09:08.208785 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:09:08.208746    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:09:28.693652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.693608    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"]
Oct 30 13:09:28.694269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.694160    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6ceae9cc-460e-472f-a47a-e78fa8b187a1" containerName="authorino"
Oct 30 13:09:28.694269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.694181    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ceae9cc-460e-472f-a47a-e78fa8b187a1" containerName="authorino"
Oct 30 13:09:28.694269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.694267    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ceae9cc-460e-472f-a47a-e78fa8b187a1" containerName="authorino"
Oct 30 13:09:28.697256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.697236    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:28.713481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.713452    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"]
Oct 30 13:09:28.811154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.811120    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt89r\" (UniqueName: \"kubernetes.io/projected/8e3b7ae4-458e-46c5-a75a-d6512feff67b-kube-api-access-gt89r\") pod \"llmisvc-controller-manager-6466c554d8-9fqwz\" (UID: \"8e3b7ae4-458e-46c5-a75a-d6512feff67b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:28.811327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.811268    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e3b7ae4-458e-46c5-a75a-d6512feff67b-cert\") pod \"llmisvc-controller-manager-6466c554d8-9fqwz\" (UID: \"8e3b7ae4-458e-46c5-a75a-d6512feff67b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:28.912279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.912242    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e3b7ae4-458e-46c5-a75a-d6512feff67b-cert\") pod \"llmisvc-controller-manager-6466c554d8-9fqwz\" (UID: \"8e3b7ae4-458e-46c5-a75a-d6512feff67b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:28.912279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.912283    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt89r\" (UniqueName: \"kubernetes.io/projected/8e3b7ae4-458e-46c5-a75a-d6512feff67b-kube-api-access-gt89r\") pod \"llmisvc-controller-manager-6466c554d8-9fqwz\" (UID: \"8e3b7ae4-458e-46c5-a75a-d6512feff67b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:28.914742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.914710    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8e3b7ae4-458e-46c5-a75a-d6512feff67b-cert\") pod \"llmisvc-controller-manager-6466c554d8-9fqwz\" (UID: \"8e3b7ae4-458e-46c5-a75a-d6512feff67b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:28.930755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:28.930724    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt89r\" (UniqueName: \"kubernetes.io/projected/8e3b7ae4-458e-46c5-a75a-d6512feff67b-kube-api-access-gt89r\") pod \"llmisvc-controller-manager-6466c554d8-9fqwz\" (UID: \"8e3b7ae4-458e-46c5-a75a-d6512feff67b\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:29.010602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:29.010512    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:29.307549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:29.307479    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"]
Oct 30 13:09:29.310526 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:09:29.310497    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e3b7ae4_458e_46c5_a75a_d6512feff67b.slice/crio-aa4e88bf55e45caea443be6c351db7d45077f9b7e6f3a1a1d8d91629cb7e8c8e WatchSource:0}: Error finding container aa4e88bf55e45caea443be6c351db7d45077f9b7e6f3a1a1d8d91629cb7e8c8e: Status 404 returned error can't find the container with id aa4e88bf55e45caea443be6c351db7d45077f9b7e6f3a1a1d8d91629cb7e8c8e
Oct 30 13:09:29.520402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:29.520366    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz" event={"ID":"8e3b7ae4-458e-46c5-a75a-d6512feff67b","Type":"ContainerStarted","Data":"aa4e88bf55e45caea443be6c351db7d45077f9b7e6f3a1a1d8d91629cb7e8c8e"}
Oct 30 13:09:30.526518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:30.526482    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz" event={"ID":"8e3b7ae4-458e-46c5-a75a-d6512feff67b","Type":"ContainerStarted","Data":"f52f093f4d40f019d4e5a21d20010ecdbf3f142b99cfefcc0b081e7b22a60d24"}
Oct 30 13:09:30.526961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:30.526738    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:09:30.565971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:09:30.565912    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz" podStartSLOduration=2.231972714 podStartE2EDuration="2.565896486s" totalImagesPullingTime="333.923772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:09:30.564875088 +0000 UTC m=+380.825113322" watchObservedRunningTime="2026-10-30 13:09:30.565896486 +0000 UTC m=+380.826134719"
Oct 30 13:09:38.209349 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:09:38.209314    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:10:01.531924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.531838    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-9fqwz"
Oct 30 13:10:01.600868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.600833    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"]
Oct 30 13:10:01.601094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.601069    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr" podUID="9bfe531a-b76a-45a5-b344-4925b5ea4426" containerName="manager" containerID="cri-o://8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f" gracePeriod=30
Oct 30 13:10:01.853026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.853001    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:10:01.893080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.893051    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bfe531a-b76a-45a5-b344-4925b5ea4426-kube-api-access-4bxfc\" (UniqueName: \"kubernetes.io/projected/9bfe531a-b76a-45a5-b344-4925b5ea4426-kube-api-access-4bxfc\") pod \"9bfe531a-b76a-45a5-b344-4925b5ea4426\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") "
Oct 30 13:10:01.893270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.893125    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bfe531a-b76a-45a5-b344-4925b5ea4426-cert\" (UniqueName: \"kubernetes.io/secret/9bfe531a-b76a-45a5-b344-4925b5ea4426-cert\") pod \"9bfe531a-b76a-45a5-b344-4925b5ea4426\" (UID: \"9bfe531a-b76a-45a5-b344-4925b5ea4426\") "
Oct 30 13:10:01.895402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.895367    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bfe531a-b76a-45a5-b344-4925b5ea4426-cert" pod "9bfe531a-b76a-45a5-b344-4925b5ea4426" (UID: "9bfe531a-b76a-45a5-b344-4925b5ea4426"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:10:01.895527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.895485    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bfe531a-b76a-45a5-b344-4925b5ea4426-kube-api-access-4bxfc" pod "9bfe531a-b76a-45a5-b344-4925b5ea4426" (UID: "9bfe531a-b76a-45a5-b344-4925b5ea4426"). InnerVolumeSpecName "kube-api-access-4bxfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:10:01.994139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.994098    4291 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9bfe531a-b76a-45a5-b344-4925b5ea4426-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:10:01.994139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:01.994137    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bxfc\" (UniqueName: \"kubernetes.io/projected/9bfe531a-b76a-45a5-b344-4925b5ea4426-kube-api-access-4bxfc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:10:02.677116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.677069    4291 generic.go:356] "Generic (PLEG): container finished" podID="9bfe531a-b76a-45a5-b344-4925b5ea4426" containerID="8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f" exitCode=0
Oct 30 13:10:02.677609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.677166    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"
Oct 30 13:10:02.677609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.677165    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr" event={"ID":"9bfe531a-b76a-45a5-b344-4925b5ea4426","Type":"ContainerDied","Data":"8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f"}
Oct 30 13:10:02.677609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.677271    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr" event={"ID":"9bfe531a-b76a-45a5-b344-4925b5ea4426","Type":"ContainerDied","Data":"91702af6891bc39ce8977d78c2f5d3c0695ab182847ddd8e9807bed87eaed419"}
Oct 30 13:10:02.677609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.677307    4291 scope.go:122] "RemoveContainer" containerID="8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f"
Oct 30 13:10:02.686530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.686503    4291 scope.go:122] "RemoveContainer" containerID="8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f"
Oct 30 13:10:02.686924 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:10:02.686897    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f\": container with ID starting with 8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f not found: ID does not exist" containerID="8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f"
Oct 30 13:10:02.687049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.686936    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f"} err="failed to get container status \"8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f\": rpc error: code = NotFound desc = could not find container \"8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f\": container with ID starting with 8585c1dd4291defc2701273d9aeaccf4d9a7de2d2fef4a4ef25ce66154b7292f not found: ID does not exist"
Oct 30 13:10:02.726035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.725994    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"]
Oct 30 13:10:02.737988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:02.737954    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-p2vxr"]
Oct 30 13:10:03.876159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:03.876125    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bfe531a-b76a-45a5-b344-4925b5ea4426" path="/var/lib/kubelet/pods/9bfe531a-b76a-45a5-b344-4925b5ea4426/volumes"
Oct 30 13:10:08.209513 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:10:08.209478    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:10:38.209012 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:10:38.208976    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:10:52.451122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:52.451079    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:10:56.444609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:56.444571    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:10:59.686155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:10:59.686113    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:11:06.181076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:06.181037    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:11:08.209966 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:11:08.209928    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:11:09.225391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.225350    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"]
Oct 30 13:11:09.226122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.226102    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9bfe531a-b76a-45a5-b344-4925b5ea4426" containerName="manager"
Oct 30 13:11:09.226212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.226125    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bfe531a-b76a-45a5-b344-4925b5ea4426" containerName="manager"
Oct 30 13:11:09.226340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.226328    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bfe531a-b76a-45a5-b344-4925b5ea4426" containerName="manager"
Oct 30 13:11:09.230619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.230593    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.256793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.256761    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.256935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.256799    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.256935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.256863    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611a92a-f72d-4bae-b03a-fa1b14777d11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.256935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.256884    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfp7\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-kube-api-access-qtfp7\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.256935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.256910    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.257087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.257020    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.257087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.257061    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.257168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.257151    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.257200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.257178    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.260501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.260476    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cw6t5\""
Oct 30 13:11:09.263301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.263275    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"]
Oct 30 13:11:09.357885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.357837    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.357885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.357895    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358016    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358087    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358118    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358169    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358204    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358252    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611a92a-f72d-4bae-b03a-fa1b14777d11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358271    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358280    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfp7\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-kube-api-access-qtfp7\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358500    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358655    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.358866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.358712    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.359121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.359090    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611a92a-f72d-4bae-b03a-fa1b14777d11-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.360714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.360678    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.360995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.360836    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.377538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.377493    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.377712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.377607    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfp7\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-kube-api-access-qtfp7\") pod \"maas-default-gateway-openshift-default-67bf986f99-svl2n\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.544765 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.544676    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:09.861438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.861399    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"]
Oct 30 13:11:09.865363 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:11:09.865337    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1611a92a_f72d_4bae_b03a_fa1b14777d11.slice/crio-f892c7212f2b5aa09ea3e13319f9f88fbddd434ab9a4ae9adfb3532d34043484 WatchSource:0}: Error finding container f892c7212f2b5aa09ea3e13319f9f88fbddd434ab9a4ae9adfb3532d34043484: Status 404 returned error can't find the container with id f892c7212f2b5aa09ea3e13319f9f88fbddd434ab9a4ae9adfb3532d34043484
Oct 30 13:11:09.868064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.868021    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:09.868165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.868114    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:09.868165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.868155    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:09.969413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.969374    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" event={"ID":"1611a92a-f72d-4bae-b03a-fa1b14777d11","Type":"ContainerStarted","Data":"1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb"}
Oct 30 13:11:09.969595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:09.969444    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" event={"ID":"1611a92a-f72d-4bae-b03a-fa1b14777d11","Type":"ContainerStarted","Data":"f892c7212f2b5aa09ea3e13319f9f88fbddd434ab9a4ae9adfb3532d34043484"}
Oct 30 13:11:10.014216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:10.014160    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podStartSLOduration=1.014142816 podStartE2EDuration="1.014142816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:10.013196723 +0000 UTC m=+480.273434956" watchObservedRunningTime="2026-10-30 13:11:10.014142816 +0000 UTC m=+480.274381050"
Oct 30 13:11:10.545487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:10.545449    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:10.547258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:10.547220    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 13:11:10.547375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:10.547300    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 13:11:11.545837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:11.545795    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 13:11:11.546351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:11.545859    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 13:11:12.545414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:12.545376    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 13:11:12.545609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:12.545433    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 13:11:13.012402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:13.012371    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:11:13.545393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:13.545356    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 13:11:13.545577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:13.545412    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 13:11:14.545289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:14.545247    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 13:11:14.545817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:14.545309    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 13:11:15.545227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:15.545195    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 13:11:15.545385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:15.545251    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 13:11:16.545228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:16.545186    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body=
Oct 30 13:11:16.545405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:16.545251    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused"
Oct 30 13:11:18.546469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:18.546430    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:18.546984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:18.546538    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:19.587898 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:19.587840    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-svl2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:11:19.588429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:19.587930    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:11:19.879086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:19.879056    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:19.879419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:19.879396    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:19.880225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:19.880208    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:11:23.931081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:23.931002    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fd7868fd-cqttk"]
Oct 30 13:11:23.931490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:23.931180    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5fd7868fd-cqttk" podUID="e226fc68-12d0-48ec-a0ed-a588f3d032f1" containerName="authorino" containerID="cri-o://a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c" gracePeriod=30
Oct 30 13:11:24.183366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.183310    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:11:24.294112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.294057    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e226fc68-12d0-48ec-a0ed-a588f3d032f1-kube-api-access-glmp7\" (UniqueName: \"kubernetes.io/projected/e226fc68-12d0-48ec-a0ed-a588f3d032f1-kube-api-access-glmp7\") pod \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") "
Oct 30 13:11:24.294281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.294150    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e226fc68-12d0-48ec-a0ed-a588f3d032f1-tls-cert\" (UniqueName: \"kubernetes.io/secret/e226fc68-12d0-48ec-a0ed-a588f3d032f1-tls-cert\") pod \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\" (UID: \"e226fc68-12d0-48ec-a0ed-a588f3d032f1\") "
Oct 30 13:11:24.296223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.296191    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e226fc68-12d0-48ec-a0ed-a588f3d032f1-kube-api-access-glmp7" pod "e226fc68-12d0-48ec-a0ed-a588f3d032f1" (UID: "e226fc68-12d0-48ec-a0ed-a588f3d032f1"). InnerVolumeSpecName "kube-api-access-glmp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:11:24.302861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.302838    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"]
Oct 30 13:11:24.303203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.303190    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e226fc68-12d0-48ec-a0ed-a588f3d032f1" containerName="authorino"
Oct 30 13:11:24.303249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.303205    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e226fc68-12d0-48ec-a0ed-a588f3d032f1" containerName="authorino"
Oct 30 13:11:24.303286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.303267    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e226fc68-12d0-48ec-a0ed-a588f3d032f1" containerName="authorino"
Oct 30 13:11:24.304193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.304165    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e226fc68-12d0-48ec-a0ed-a588f3d032f1-tls-cert" pod "e226fc68-12d0-48ec-a0ed-a588f3d032f1" (UID: "e226fc68-12d0-48ec-a0ed-a588f3d032f1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:11:24.306142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.306124    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.328721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.328692    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"]
Oct 30 13:11:24.395259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395224    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395263    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395291    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dsgw\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-kube-api-access-8dsgw\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395431    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395524    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395564    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395686    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395716    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395778    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.395890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395833    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glmp7\" (UniqueName: \"kubernetes.io/projected/e226fc68-12d0-48ec-a0ed-a588f3d032f1-kube-api-access-glmp7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:11:24.395957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.395915    4291 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e226fc68-12d0-48ec-a0ed-a588f3d032f1-tls-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:11:24.496936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.496853    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.496936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.496889    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.496936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.496927    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.496949    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.496969    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.496985    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dsgw\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-kube-api-access-8dsgw\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497001    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497045    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497088    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497422    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497789    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497816    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497822    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.497910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.497837    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.499256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.499234    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.499568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.499552    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.529289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.529251    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.529434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.529362    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dsgw\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-kube-api-access-8dsgw\") pod \"maas-default-gateway-openshift-default-67bf986f99-xb65f\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.623200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.623163    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:24.944370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.944339    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"]
Oct 30 13:11:24.947311 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:11:24.947282    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88650bd5_e6e2_496b_a2c8_f58b9624eafb.slice/crio-2a22e7f094062a22f742f6b43d217647bc759abc5e22287e36fbe075f7789c1f WatchSource:0}: Error finding container 2a22e7f094062a22f742f6b43d217647bc759abc5e22287e36fbe075f7789c1f: Status 404 returned error can't find the container with id 2a22e7f094062a22f742f6b43d217647bc759abc5e22287e36fbe075f7789c1f
Oct 30 13:11:24.949444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.949414    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:24.949543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.949477    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:24.949543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:24.949506    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:25.042956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.042919    4291 generic.go:356] "Generic (PLEG): container finished" podID="e226fc68-12d0-48ec-a0ed-a588f3d032f1" containerID="a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c" exitCode=0
Oct 30 13:11:25.043135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.042966    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5fd7868fd-cqttk"
Oct 30 13:11:25.043135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.043002    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd7868fd-cqttk" event={"ID":"e226fc68-12d0-48ec-a0ed-a588f3d032f1","Type":"ContainerDied","Data":"a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c"}
Oct 30 13:11:25.043135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.043042    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5fd7868fd-cqttk" event={"ID":"e226fc68-12d0-48ec-a0ed-a588f3d032f1","Type":"ContainerDied","Data":"06625ff8920d265d64e403b178d5891ef112c93e7439696f47531436819f9728"}
Oct 30 13:11:25.043135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.043063    4291 scope.go:122] "RemoveContainer" containerID="a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c"
Oct 30 13:11:25.044799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.044762    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" event={"ID":"88650bd5-e6e2-496b-a2c8-f58b9624eafb","Type":"ContainerStarted","Data":"6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd"}
Oct 30 13:11:25.044933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.044813    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" event={"ID":"88650bd5-e6e2-496b-a2c8-f58b9624eafb","Type":"ContainerStarted","Data":"2a22e7f094062a22f742f6b43d217647bc759abc5e22287e36fbe075f7789c1f"}
Oct 30 13:11:25.054125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.054102    4291 scope.go:122] "RemoveContainer" containerID="a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c"
Oct 30 13:11:25.054428 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:11:25.054410    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c\": container with ID starting with a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c not found: ID does not exist" containerID="a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c"
Oct 30 13:11:25.054503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.054435    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c"} err="failed to get container status \"a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c\": rpc error: code = NotFound desc = could not find container \"a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c\": container with ID starting with a3eeb3aa53ebda2b764b7935f3394ac146bba6f7129ab79b6d794f3549f1bf9c not found: ID does not exist"
Oct 30 13:11:25.112113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.112063    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podStartSLOduration=1.112045988 podStartE2EDuration="1.112045988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:25.110701918 +0000 UTC m=+495.370940155" watchObservedRunningTime="2026-10-30 13:11:25.112045988 +0000 UTC m=+495.372284219"
Oct 30 13:11:25.146987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.146953    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5fd7868fd-cqttk"]
Oct 30 13:11:25.155079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.155053    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5fd7868fd-cqttk"]
Oct 30 13:11:25.624203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.624168    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:25.880236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:25.880128    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e226fc68-12d0-48ec-a0ed-a588f3d032f1" path="/var/lib/kubelet/pods/e226fc68-12d0-48ec-a0ed-a588f3d032f1/volumes"
Oct 30 13:11:26.625620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:26.625584    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xb65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 13:11:26.626027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:26.625664    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 13:11:27.624188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:27.624154    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xb65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 13:11:27.624341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:27.624210    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 13:11:28.624584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:28.624546    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xb65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 13:11:28.625004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:28.624601    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 13:11:29.623883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:29.623845    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xb65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 13:11:29.624061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:29.623904    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 13:11:30.624092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:30.624055    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xb65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 13:11:30.624483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:30.624106    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 13:11:31.624066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:31.624019    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xb65f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body=
Oct 30 13:11:31.624246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:31.624092    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused"
Oct 30 13:11:33.179415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:33.179384    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:33.179872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:33.179725    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:33.180513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:33.180485    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:11:38.209010 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:11:38.208960    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:11:39.340599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.340568    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"]
Oct 30 13:11:39.345507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.345486    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.373803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.373774    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"]
Oct 30 13:11:39.461037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.460998    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461037    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461067    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461216    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461245    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461269    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461284    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461327    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.461500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.461478    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh54j\" (UniqueName: \"kubernetes.io/projected/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-kube-api-access-hh54j\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.562468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562428    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.562468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562472    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.562775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562517    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.562775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562537    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.562775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562559    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.562775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562599    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh54j\" (UniqueName: \"kubernetes.io/projected/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-kube-api-access-hh54j\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.562962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562819    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.563057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.563037    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.563093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.563049    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.563093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.563073    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.563093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562984    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.563204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.562945    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.563204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.563076    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.563322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.563305    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.565286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.565253    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.565388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.565362    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.582591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.582566    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.582899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.582879    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh54j\" (UniqueName: \"kubernetes.io/projected/da312528-eb75-4d61-89fb-3c2a6ca8c6aa-kube-api-access-hh54j\") pod \"maas-default-gateway-openshift-default-67bf986f99-zlnl2\" (UID: \"da312528-eb75-4d61-89fb-3c2a6ca8c6aa\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.657837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.657801    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:39.976343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.976315    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"]
Oct 30 13:11:39.978232 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:11:39.978204    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda312528_eb75_4d61_89fb_3c2a6ca8c6aa.slice/crio-2ea89ddb1db6aa78585244b52a49051b4edc3ea6b8b1043f75c1954d456b1a42 WatchSource:0}: Error finding container 2ea89ddb1db6aa78585244b52a49051b4edc3ea6b8b1043f75c1954d456b1a42: Status 404 returned error can't find the container with id 2ea89ddb1db6aa78585244b52a49051b4edc3ea6b8b1043f75c1954d456b1a42
Oct 30 13:11:39.980355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.980326    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:39.980424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.980391    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:39.980424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:39.980418    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:11:40.137348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:40.137317    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" event={"ID":"da312528-eb75-4d61-89fb-3c2a6ca8c6aa","Type":"ContainerStarted","Data":"d3dbb46e162726df6ab16a13ab5066d5618b31b625d84bc261ac0525b588958c"}
Oct 30 13:11:40.137585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:40.137368    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" event={"ID":"da312528-eb75-4d61-89fb-3c2a6ca8c6aa","Type":"ContainerStarted","Data":"2ea89ddb1db6aa78585244b52a49051b4edc3ea6b8b1043f75c1954d456b1a42"}
Oct 30 13:11:40.181707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:40.181661    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podStartSLOduration=1.18162117 podStartE2EDuration="1.18162117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:11:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:11:40.180837482 +0000 UTC m=+510.441075717" watchObservedRunningTime="2026-10-30 13:11:40.18162117 +0000 UTC m=+510.441859443"
Oct 30 13:11:40.665821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:40.665727    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:41.666565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:41.666527    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 13:11:41.667180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:41.667145    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 13:11:42.665084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:42.665042    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 13:11:42.665386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:42.665355    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 13:11:43.667932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:43.667895    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 13:11:43.668709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:43.668676    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 13:11:44.666065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:44.666025    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 13:11:44.666374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:44.666338    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 13:11:45.666060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:45.666026    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 13:11:45.666678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:45.666616    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 13:11:46.658713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:46.658676    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Oct 30 13:11:46.658889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:46.658729    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Oct 30 13:11:48.166100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:48.166067    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:48.166560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:48.166406    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:11:48.167168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:11:48.167149    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2"
Oct 30 13:12:01.800668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:01.800591    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:12:08.212368 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:12:08.212314    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:12:12.291026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.290985    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:12:12.889954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.889918    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"]
Oct 30 13:12:12.894548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.894527    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:12.908728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.908695    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"]
Oct 30 13:12:12.918852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.918820    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:12:12.919040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.918884    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:12:12.919040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.918969    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:12:12.920623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.920604    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:12:12.966865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.966826    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:12.967054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.966884    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sspcx\" (UniqueName: \"kubernetes.io/projected/d967dd86-6fc3-40aa-8da4-a808aba71cb2-kube-api-access-sspcx\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:12.967054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:12.966993    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.068376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.068327    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.068587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.068425    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.068587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.068473    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sspcx\" (UniqueName: \"kubernetes.io/projected/d967dd86-6fc3-40aa-8da4-a808aba71cb2-kube-api-access-sspcx\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.069024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.069000    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.070980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.070951    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.087562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.087522    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sspcx\" (UniqueName: \"kubernetes.io/projected/d967dd86-6fc3-40aa-8da4-a808aba71cb2-kube-api-access-sspcx\") pod \"maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.208551 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.208466    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:13.510074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:13.510048    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"]
Oct 30 13:12:13.512768 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:12:13.512732    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd967dd86_6fc3_40aa_8da4_a808aba71cb2.slice/crio-1e414fcf8a0eeeea057d13520ce153ffdb5c5d85642352ab729c0625d25051c3 WatchSource:0}: Error finding container 1e414fcf8a0eeeea057d13520ce153ffdb5c5d85642352ab729c0625d25051c3: Status 404 returned error can't find the container with id 1e414fcf8a0eeeea057d13520ce153ffdb5c5d85642352ab729c0625d25051c3
Oct 30 13:12:14.316093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:14.316049    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj" event={"ID":"d967dd86-6fc3-40aa-8da4-a808aba71cb2","Type":"ContainerStarted","Data":"1e414fcf8a0eeeea057d13520ce153ffdb5c5d85642352ab729c0625d25051c3"}
Oct 30 13:12:16.328101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:16.328014    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj" event={"ID":"d967dd86-6fc3-40aa-8da4-a808aba71cb2","Type":"ContainerStarted","Data":"7d1f5599ed55cd4d5c1766e55386b9ffe2d0ba9c8271db86910e75c6215492a9"}
Oct 30 13:12:16.369805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:16.369712    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj" podStartSLOduration=2.082756631 podStartE2EDuration="4.369694538s" totalImagesPullingTime="2.286937907s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:16.368124142 +0000 UTC m=+546.628362376" watchObservedRunningTime="2026-10-30 13:12:16.369694538 +0000 UTC m=+546.629932766"
Oct 30 13:12:17.334806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:17.334771    4291 generic.go:356] "Generic (PLEG): container finished" podID="d967dd86-6fc3-40aa-8da4-a808aba71cb2" containerID="7d1f5599ed55cd4d5c1766e55386b9ffe2d0ba9c8271db86910e75c6215492a9" exitCode=0
Oct 30 13:12:17.335217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:17.334832    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj" event={"ID":"d967dd86-6fc3-40aa-8da4-a808aba71cb2","Type":"ContainerDied","Data":"7d1f5599ed55cd4d5c1766e55386b9ffe2d0ba9c8271db86910e75c6215492a9"}
Oct 30 13:12:18.146244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.146208    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"]
Oct 30 13:12:18.149891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.149873    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.168053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.168024    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"]
Oct 30 13:12:18.218723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.218687    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgjpm\" (UniqueName: \"kubernetes.io/projected/f1d8347f-8706-4b80-9871-5d0160ad5a81-kube-api-access-xgjpm\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.218899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.218733    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.218899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.218798    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.319849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.319812    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.320034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.319863    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.320034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.319932    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgjpm\" (UniqueName: \"kubernetes.io/projected/f1d8347f-8706-4b80-9871-5d0160ad5a81-kube-api-access-xgjpm\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.320623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.320602    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.322372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.322348    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.338956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.338929    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgjpm\" (UniqueName: \"kubernetes.io/projected/f1d8347f-8706-4b80-9871-5d0160ad5a81-kube-api-access-xgjpm\") pod \"maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.464707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.463813    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:18.515390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.515362    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:18.624526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.624469    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-openshift-ca\") pod \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") "
Oct 30 13:12:18.624777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.624561    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-xks-ca\") pod \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") "
Oct 30 13:12:18.624777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.624723    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d967dd86-6fc3-40aa-8da4-a808aba71cb2-kube-api-access-sspcx\" (UniqueName: \"kubernetes.io/projected/d967dd86-6fc3-40aa-8da4-a808aba71cb2-kube-api-access-sspcx\") pod \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\" (UID: \"d967dd86-6fc3-40aa-8da4-a808aba71cb2\") "
Oct 30 13:12:18.625015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.624987    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-openshift-ca" pod "d967dd86-6fc3-40aa-8da4-a808aba71cb2" (UID: "d967dd86-6fc3-40aa-8da4-a808aba71cb2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:18.625116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.625089    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:18.627330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.627296    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d967dd86-6fc3-40aa-8da4-a808aba71cb2-kube-api-access-sspcx" pod "d967dd86-6fc3-40aa-8da4-a808aba71cb2" (UID: "d967dd86-6fc3-40aa-8da4-a808aba71cb2"). InnerVolumeSpecName "kube-api-access-sspcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:18.627454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.627298    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-xks-ca" pod "d967dd86-6fc3-40aa-8da4-a808aba71cb2" (UID: "d967dd86-6fc3-40aa-8da4-a808aba71cb2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:18.726592 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.726507    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d967dd86-6fc3-40aa-8da4-a808aba71cb2-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:18.726592 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.726536    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sspcx\" (UniqueName: \"kubernetes.io/projected/d967dd86-6fc3-40aa-8da4-a808aba71cb2-kube-api-access-sspcx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:18.785524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:18.785485    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"]
Oct 30 13:12:18.788976 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:12:18.788936    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d8347f_8706_4b80_9871_5d0160ad5a81.slice/crio-eea4d47f2c33d86964de634a2cd55ee952e85d0f22586f0e5f83e1bd4173219f WatchSource:0}: Error finding container eea4d47f2c33d86964de634a2cd55ee952e85d0f22586f0e5f83e1bd4173219f: Status 404 returned error can't find the container with id eea4d47f2c33d86964de634a2cd55ee952e85d0f22586f0e5f83e1bd4173219f
Oct 30 13:12:19.348549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:19.348507    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l" event={"ID":"f1d8347f-8706-4b80-9871-5d0160ad5a81","Type":"ContainerStarted","Data":"6a54819ffd60b9e618dba0114fd2e49b8bb9a6574ee1adbe6f684549e4ad2fae"}
Oct 30 13:12:19.348549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:19.348567    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l" event={"ID":"f1d8347f-8706-4b80-9871-5d0160ad5a81","Type":"ContainerStarted","Data":"eea4d47f2c33d86964de634a2cd55ee952e85d0f22586f0e5f83e1bd4173219f"}
Oct 30 13:12:19.349840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:19.349817    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"
Oct 30 13:12:19.349969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:19.349831    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj" event={"ID":"d967dd86-6fc3-40aa-8da4-a808aba71cb2","Type":"ContainerDied","Data":"1e414fcf8a0eeeea057d13520ce153ffdb5c5d85642352ab729c0625d25051c3"}
Oct 30 13:12:19.349969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:19.349873    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e414fcf8a0eeeea057d13520ce153ffdb5c5d85642352ab729c0625d25051c3"
Oct 30 13:12:19.386038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:19.385987    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l" podStartSLOduration=1.385970611 podStartE2EDuration="1.385970611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:19.385224188 +0000 UTC m=+549.645462422" watchObservedRunningTime="2026-10-30 13:12:19.385970611 +0000 UTC m=+549.646208848"
Oct 30 13:12:20.192856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:20.192809    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:12:20.355546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:20.355514    4291 generic.go:356] "Generic (PLEG): container finished" podID="f1d8347f-8706-4b80-9871-5d0160ad5a81" containerID="6a54819ffd60b9e618dba0114fd2e49b8bb9a6574ee1adbe6f684549e4ad2fae" exitCode=0
Oct 30 13:12:20.355959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:20.355606    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l" event={"ID":"f1d8347f-8706-4b80-9871-5d0160ad5a81","Type":"ContainerDied","Data":"6a54819ffd60b9e618dba0114fd2e49b8bb9a6574ee1adbe6f684549e4ad2fae"}
Oct 30 13:12:21.497112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.497081    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:21.656123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.656024    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1d8347f-8706-4b80-9871-5d0160ad5a81-kube-api-access-xgjpm\" (UniqueName: \"kubernetes.io/projected/f1d8347f-8706-4b80-9871-5d0160ad5a81-kube-api-access-xgjpm\") pod \"f1d8347f-8706-4b80-9871-5d0160ad5a81\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") "
Oct 30 13:12:21.656123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.656082    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-xks-ca\") pod \"f1d8347f-8706-4b80-9871-5d0160ad5a81\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") "
Oct 30 13:12:21.656123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.656120    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-openshift-ca\") pod \"f1d8347f-8706-4b80-9871-5d0160ad5a81\" (UID: \"f1d8347f-8706-4b80-9871-5d0160ad5a81\") "
Oct 30 13:12:21.656530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.656506    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-openshift-ca" pod "f1d8347f-8706-4b80-9871-5d0160ad5a81" (UID: "f1d8347f-8706-4b80-9871-5d0160ad5a81"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:21.658316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.658284    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-xks-ca" pod "f1d8347f-8706-4b80-9871-5d0160ad5a81" (UID: "f1d8347f-8706-4b80-9871-5d0160ad5a81"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:21.658316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.658302    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1d8347f-8706-4b80-9871-5d0160ad5a81-kube-api-access-xgjpm" pod "f1d8347f-8706-4b80-9871-5d0160ad5a81" (UID: "f1d8347f-8706-4b80-9871-5d0160ad5a81"). InnerVolumeSpecName "kube-api-access-xgjpm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:21.757750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.757712    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgjpm\" (UniqueName: \"kubernetes.io/projected/f1d8347f-8706-4b80-9871-5d0160ad5a81-kube-api-access-xgjpm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:21.757750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.757744    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:21.757750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:21.757754    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f1d8347f-8706-4b80-9871-5d0160ad5a81-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:22.366067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:22.366034    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"
Oct 30 13:12:22.366067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:22.366042    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l" event={"ID":"f1d8347f-8706-4b80-9871-5d0160ad5a81","Type":"ContainerDied","Data":"eea4d47f2c33d86964de634a2cd55ee952e85d0f22586f0e5f83e1bd4173219f"}
Oct 30 13:12:22.366301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:22.366089    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eea4d47f2c33d86964de634a2cd55ee952e85d0f22586f0e5f83e1bd4173219f"
Oct 30 13:12:25.616955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.616923    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"]
Oct 30 13:12:25.617383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.617294    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d967dd86-6fc3-40aa-8da4-a808aba71cb2" containerName="revoke-keys"
Oct 30 13:12:25.617383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.617305    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d967dd86-6fc3-40aa-8da4-a808aba71cb2" containerName="revoke-keys"
Oct 30 13:12:25.617383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.617314    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f1d8347f-8706-4b80-9871-5d0160ad5a81" containerName="revoke-keys"
Oct 30 13:12:25.617383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.617319    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1d8347f-8706-4b80-9871-5d0160ad5a81" containerName="revoke-keys"
Oct 30 13:12:25.617383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.617384    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d967dd86-6fc3-40aa-8da4-a808aba71cb2" containerName="revoke-keys"
Oct 30 13:12:25.617536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.617392    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1d8347f-8706-4b80-9871-5d0160ad5a81" containerName="revoke-keys"
Oct 30 13:12:25.620469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.620448    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.641120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.641083    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"]
Oct 30 13:12:25.647730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.647695    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:12:25.647942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.647768    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:12:25.647942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.647803    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:12:25.647942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.647813    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:12:25.796739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.796700    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwm8\" (UniqueName: \"kubernetes.io/projected/21ef08ad-dab5-4b71-927b-ded0881e4d19-kube-api-access-bfwm8\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.796939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.796751    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.796939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.796838    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.897625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.897523    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwm8\" (UniqueName: \"kubernetes.io/projected/21ef08ad-dab5-4b71-927b-ded0881e4d19-kube-api-access-bfwm8\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.897625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.897572    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.897625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.897601    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.898244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.898223    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.899972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.899944    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.918130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.918097    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwm8\" (UniqueName: \"kubernetes.io/projected/21ef08ad-dab5-4b71-927b-ded0881e4d19-kube-api-access-bfwm8\") pod \"maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:25.933103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:25.933067    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:26.241065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:26.241036    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"]
Oct 30 13:12:26.249119 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:12:26.249089    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ef08ad_dab5_4b71_927b_ded0881e4d19.slice/crio-2b9cdf5a5eebcf44cd6fa1638b79de3110114d8994e409a85e0b497d32ccc433 WatchSource:0}: Error finding container 2b9cdf5a5eebcf44cd6fa1638b79de3110114d8994e409a85e0b497d32ccc433: Status 404 returned error can't find the container with id 2b9cdf5a5eebcf44cd6fa1638b79de3110114d8994e409a85e0b497d32ccc433
Oct 30 13:12:26.384275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:26.384237    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r" event={"ID":"21ef08ad-dab5-4b71-927b-ded0881e4d19","Type":"ContainerStarted","Data":"2b9cdf5a5eebcf44cd6fa1638b79de3110114d8994e409a85e0b497d32ccc433"}
Oct 30 13:12:27.390150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:27.390119    4291 generic.go:356] "Generic (PLEG): container finished" podID="21ef08ad-dab5-4b71-927b-ded0881e4d19" containerID="7f1ec032fe417d8027ea9b3b4d2f8e71bf68373853fc9f1c200f13944cb93fb1" exitCode=0
Oct 30 13:12:27.390682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:27.390205    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r" event={"ID":"21ef08ad-dab5-4b71-927b-ded0881e4d19","Type":"ContainerDied","Data":"7f1ec032fe417d8027ea9b3b4d2f8e71bf68373853fc9f1c200f13944cb93fb1"}
Oct 30 13:12:28.530835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.530809    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:28.622694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.622662    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21ef08ad-dab5-4b71-927b-ded0881e4d19-kube-api-access-bfwm8\" (UniqueName: \"kubernetes.io/projected/21ef08ad-dab5-4b71-927b-ded0881e4d19-kube-api-access-bfwm8\") pod \"21ef08ad-dab5-4b71-927b-ded0881e4d19\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") "
Oct 30 13:12:28.622870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.622706    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-xks-ca\") pod \"21ef08ad-dab5-4b71-927b-ded0881e4d19\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") "
Oct 30 13:12:28.622870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.622828    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-openshift-ca\") pod \"21ef08ad-dab5-4b71-927b-ded0881e4d19\" (UID: \"21ef08ad-dab5-4b71-927b-ded0881e4d19\") "
Oct 30 13:12:28.623168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.623140    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-openshift-ca" pod "21ef08ad-dab5-4b71-927b-ded0881e4d19" (UID: "21ef08ad-dab5-4b71-927b-ded0881e4d19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:28.624821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.624791    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-xks-ca" pod "21ef08ad-dab5-4b71-927b-ded0881e4d19" (UID: "21ef08ad-dab5-4b71-927b-ded0881e4d19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:28.624909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.624828    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21ef08ad-dab5-4b71-927b-ded0881e4d19-kube-api-access-bfwm8" pod "21ef08ad-dab5-4b71-927b-ded0881e4d19" (UID: "21ef08ad-dab5-4b71-927b-ded0881e4d19"). InnerVolumeSpecName "kube-api-access-bfwm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:28.724527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.724440    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.724527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.724473    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21ef08ad-dab5-4b71-927b-ded0881e4d19-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.724527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.724483    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwm8\" (UniqueName: \"kubernetes.io/projected/21ef08ad-dab5-4b71-927b-ded0881e4d19-kube-api-access-bfwm8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:28.830384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.830346    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"]
Oct 30 13:12:28.830768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.830754    4291 cpu_manager.go:414] "containerMap: removing container" podUID="21ef08ad-dab5-4b71-927b-ded0881e4d19" containerName="revoke-keys"
Oct 30 13:12:28.830817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.830770    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21ef08ad-dab5-4b71-927b-ded0881e4d19" containerName="revoke-keys"
Oct 30 13:12:28.830873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.830862    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="21ef08ad-dab5-4b71-927b-ded0881e4d19" containerName="revoke-keys"
Oct 30 13:12:28.833878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.833855    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:28.850733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.850699    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"]
Oct 30 13:12:28.927019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.926979    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:28.927219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.927029    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:28.927219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:28.927172    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhfgl\" (UniqueName: \"kubernetes.io/projected/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-kube-api-access-jhfgl\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.027906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.027796    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhfgl\" (UniqueName: \"kubernetes.io/projected/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-kube-api-access-jhfgl\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.027906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.027882    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.028138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.027932    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.028863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.028834    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.031045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.031014    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.047855 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.047823    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhfgl\" (UniqueName: \"kubernetes.io/projected/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-kube-api-access-jhfgl\") pod \"maas-api-revoke-keys-e2e-ait-662ed421-zhm7n\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.146649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.146601    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:29.401348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.401321    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"
Oct 30 13:12:29.401348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.401328    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r" event={"ID":"21ef08ad-dab5-4b71-927b-ded0881e4d19","Type":"ContainerDied","Data":"2b9cdf5a5eebcf44cd6fa1638b79de3110114d8994e409a85e0b497d32ccc433"}
Oct 30 13:12:29.401542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.401375    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b9cdf5a5eebcf44cd6fa1638b79de3110114d8994e409a85e0b497d32ccc433"
Oct 30 13:12:29.445953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:29.445923    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"]
Oct 30 13:12:29.448944 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:12:29.448913    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f730cb6_6dc8_4bda_a0b2_93cb4d1aac4b.slice/crio-7fb47b1923de7d3bc33fd2b51e1e5b1634d73d4ec3ba845a8befd3526969c06b WatchSource:0}: Error finding container 7fb47b1923de7d3bc33fd2b51e1e5b1634d73d4ec3ba845a8befd3526969c06b: Status 404 returned error can't find the container with id 7fb47b1923de7d3bc33fd2b51e1e5b1634d73d4ec3ba845a8befd3526969c06b
Oct 30 13:12:30.407269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:30.407234    4291 generic.go:356] "Generic (PLEG): container finished" podID="1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" containerID="a8c1f06c7828835b08589c9012356eeb47e393970b12a9732dd0b7fe46b47dd0" exitCode=0
Oct 30 13:12:30.407679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:30.407319    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n" event={"ID":"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b","Type":"ContainerDied","Data":"a8c1f06c7828835b08589c9012356eeb47e393970b12a9732dd0b7fe46b47dd0"}
Oct 30 13:12:30.407679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:30.407372    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n" event={"ID":"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b","Type":"ContainerStarted","Data":"7fb47b1923de7d3bc33fd2b51e1e5b1634d73d4ec3ba845a8befd3526969c06b"}
Oct 30 13:12:31.545035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.545010    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:31.652007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.651966    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-openshift-ca\") pod \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") "
Oct 30 13:12:31.652193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.652059    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-xks-ca\") pod \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") "
Oct 30 13:12:31.652193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.652120    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-kube-api-access-jhfgl\" (UniqueName: \"kubernetes.io/projected/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-kube-api-access-jhfgl\") pod \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\" (UID: \"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b\") "
Oct 30 13:12:31.652406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.652381    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-openshift-ca" pod "1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" (UID: "1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:31.654198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.654172    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-kube-api-access-jhfgl" pod "1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" (UID: "1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b"). InnerVolumeSpecName "kube-api-access-jhfgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:31.654263 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.654216    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-xks-ca" pod "1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" (UID: "1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:31.753214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.753118    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:31.753214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.753162    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:31.753214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:31.753174    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhfgl\" (UniqueName: \"kubernetes.io/projected/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b-kube-api-access-jhfgl\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:32.121749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.121713    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"]
Oct 30 13:12:32.122146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.122127    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" containerName="revoke-keys"
Oct 30 13:12:32.122231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.122148    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" containerName="revoke-keys"
Oct 30 13:12:32.122290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.122244    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" containerName="revoke-keys"
Oct 30 13:12:32.125339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.125315    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.153908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.153881    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"]
Oct 30 13:12:32.259356 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.259323    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.259523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.259454    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgwm2\" (UniqueName: \"kubernetes.io/projected/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-kube-api-access-bgwm2\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.259523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.259488    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.360141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.360106    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgwm2\" (UniqueName: \"kubernetes.io/projected/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-kube-api-access-bgwm2\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.360141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.360145    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.360337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.360269    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.360877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.360861    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.362563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.362546    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.379761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.379707    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgwm2\" (UniqueName: \"kubernetes.io/projected/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-kube-api-access-bgwm2\") pod \"maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.420186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.420154    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n" event={"ID":"1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b","Type":"ContainerDied","Data":"7fb47b1923de7d3bc33fd2b51e1e5b1634d73d4ec3ba845a8befd3526969c06b"}
Oct 30 13:12:32.420325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.420202    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7fb47b1923de7d3bc33fd2b51e1e5b1634d73d4ec3ba845a8befd3526969c06b"
Oct 30 13:12:32.420325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.420166    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"
Oct 30 13:12:32.436965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.436943    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:32.736439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:32.736418    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"]
Oct 30 13:12:32.738743 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:12:32.738717    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf1ce19d_1bcc_4fa7_af3b_b961781c3b29.slice/crio-312af192e8f9cb5bef573f8916506e564eaaec1114328fee33ffe4cc08fde340 WatchSource:0}: Error finding container 312af192e8f9cb5bef573f8916506e564eaaec1114328fee33ffe4cc08fde340: Status 404 returned error can't find the container with id 312af192e8f9cb5bef573f8916506e564eaaec1114328fee33ffe4cc08fde340
Oct 30 13:12:33.427065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:33.427023    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p" event={"ID":"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29","Type":"ContainerStarted","Data":"69b0ed4570ce76109a824a78f2d8383c7925fdddd544541e70e6c201b1be36c7"}
Oct 30 13:12:33.427241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:33.427090    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p" event={"ID":"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29","Type":"ContainerStarted","Data":"312af192e8f9cb5bef573f8916506e564eaaec1114328fee33ffe4cc08fde340"}
Oct 30 13:12:33.467473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:33.467418    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p" podStartSLOduration=1.467404849 podStartE2EDuration="1.467404849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:33.466882123 +0000 UTC m=+563.727120357" watchObservedRunningTime="2026-10-30 13:12:33.467404849 +0000 UTC m=+563.727643082"
Oct 30 13:12:33.609291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:33.609259    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"]
Oct 30 13:12:33.611124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:33.611102    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-d6a10701-t4xhj"]
Oct 30 13:12:33.883619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:33.882048    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d967dd86-6fc3-40aa-8da4-a808aba71cb2" path="/var/lib/kubelet/pods/d967dd86-6fc3-40aa-8da4-a808aba71cb2/volumes"
Oct 30 13:12:34.433623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:34.433584    4291 generic.go:356] "Generic (PLEG): container finished" podID="bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" containerID="69b0ed4570ce76109a824a78f2d8383c7925fdddd544541e70e6c201b1be36c7" exitCode=0
Oct 30 13:12:34.433827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:34.433666    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p" event={"ID":"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29","Type":"ContainerDied","Data":"69b0ed4570ce76109a824a78f2d8383c7925fdddd544541e70e6c201b1be36c7"}
Oct 30 13:12:35.064916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.064810    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"]
Oct 30 13:12:35.096725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.069418    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.099928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.099898    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"]
Oct 30 13:12:35.108870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.108834    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5852ff-openshift-default-dockercfg-76pkk\""
Oct 30 13:12:35.185087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185045    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185111    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185230    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185261    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185301    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3603ce-72c0-4705-9aee-66df4fd80a25-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185361    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185411    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.185672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.185451    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5wnd\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-kube-api-access-b5wnd\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.286549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286508    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.286549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286567    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3603ce-72c0-4705-9aee-66df4fd80a25-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.286846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286585    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.286846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286601    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.286846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286616    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wnd\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-kube-api-access-b5wnd\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.286846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286684    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286955    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.286992    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.287039    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.287167    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.287348    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.287380    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.287413    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3603ce-72c0-4705-9aee-66df4fd80a25-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.287614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.287595    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.289120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.289091    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.289320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.289292    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.305903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.305868    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.306082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.305953    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wnd\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-kube-api-access-b5wnd\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.386894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.386844    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:35.605881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.605852    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:35.703865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.703840    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"]
Oct 30 13:12:35.706451 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:12:35.706424    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce3603ce_72c0_4705_9aee_66df4fd80a25.slice/crio-b49065761f6037c6ff75496031de8a76b01398c99658608dda756e2e8861173d WatchSource:0}: Error finding container b49065761f6037c6ff75496031de8a76b01398c99658608dda756e2e8861173d: Status 404 returned error can't find the container with id b49065761f6037c6ff75496031de8a76b01398c99658608dda756e2e8861173d
Oct 30 13:12:35.708740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.708706    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:35.708834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.708772    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:35.708834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.708800    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:35.797221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.797188    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-xks-ca\") pod \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") "
Oct 30 13:12:35.797415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.797265    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-openshift-ca\") pod \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") "
Oct 30 13:12:35.797415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.797290    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-kube-api-access-bgwm2\" (UniqueName: \"kubernetes.io/projected/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-kube-api-access-bgwm2\") pod \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\" (UID: \"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29\") "
Oct 30 13:12:35.797696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.797657    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-openshift-ca" pod "bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" (UID: "bf1ce19d-1bcc-4fa7-af3b-b961781c3b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:12:35.799516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.799484    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-xks-ca" pod "bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" (UID: "bf1ce19d-1bcc-4fa7-af3b-b961781c3b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:12:35.799979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.799936    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-kube-api-access-bgwm2" pod "bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" (UID: "bf1ce19d-1bcc-4fa7-af3b-b961781c3b29"). InnerVolumeSpecName "kube-api-access-bgwm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:12:35.898846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.898812    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:35.898846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.898840    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:35.898846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:35.898851    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgwm2\" (UniqueName: \"kubernetes.io/projected/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29-kube-api-access-bgwm2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:12:36.008749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.008653    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"]
Oct 30 13:12:36.012245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.012205    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-fc4197ef-n5z5l"]
Oct 30 13:12:36.446018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.445987    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"
Oct 30 13:12:36.446523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.445986    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p" event={"ID":"bf1ce19d-1bcc-4fa7-af3b-b961781c3b29","Type":"ContainerDied","Data":"312af192e8f9cb5bef573f8916506e564eaaec1114328fee33ffe4cc08fde340"}
Oct 30 13:12:36.446523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.446109    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="312af192e8f9cb5bef573f8916506e564eaaec1114328fee33ffe4cc08fde340"
Oct 30 13:12:36.447837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.447810    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" event={"ID":"ce3603ce-72c0-4705-9aee-66df4fd80a25","Type":"ContainerStarted","Data":"1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913"}
Oct 30 13:12:36.447976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.447861    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" event={"ID":"ce3603ce-72c0-4705-9aee-66df4fd80a25","Type":"ContainerStarted","Data":"b49065761f6037c6ff75496031de8a76b01398c99658608dda756e2e8861173d"}
Oct 30 13:12:36.497898 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.497841    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" podStartSLOduration=1.49782514 podStartE2EDuration="1.49782514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:12:36.497049331 +0000 UTC m=+566.757287567" watchObservedRunningTime="2026-10-30 13:12:36.49782514 +0000 UTC m=+566.758063370"
Oct 30 13:12:36.974104 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:36.974069    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:12:37.387825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:37.387785    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:37.393676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:37.393630    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:37.453046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:37.452992    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:37.454055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:37.454035    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:12:37.876841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:37.876811    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1d8347f-8706-4b80-9871-5d0160ad5a81" path="/var/lib/kubelet/pods/f1d8347f-8706-4b80-9871-5d0160ad5a81/volumes"
Oct 30 13:12:38.210335 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:12:38.210243    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:12:40.181317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:40.181282    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"]
Oct 30 13:12:40.183416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:40.183393    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-deb5d83f-5lm8r"]
Oct 30 13:12:41.876388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:41.876357    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21ef08ad-dab5-4b71-927b-ded0881e4d19" path="/var/lib/kubelet/pods/21ef08ad-dab5-4b71-927b-ded0881e4d19/volumes"
Oct 30 13:12:42.576017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:42.575980    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:12:46.661121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:46.661087    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"]
Oct 30 13:12:46.673433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:46.673403    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-662ed421-zhm7n"]
Oct 30 13:12:47.793418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:47.793386    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:12:47.882386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:47.882353    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b" path="/var/lib/kubelet/pods/1f730cb6-6dc8-4bda-a0b2-93cb4d1aac4b/volumes"
Oct 30 13:12:50.802526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:50.802494    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"]
Oct 30 13:12:50.811422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:50.811398    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-fb3b54cb-p4l8p"]
Oct 30 13:12:51.884338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:51.884300    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" path="/var/lib/kubelet/pods/bf1ce19d-1bcc-4fa7-af3b-b961781c3b29/volumes"
Oct 30 13:12:52.891163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:52.891087    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:12:58.549420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:58.549388    4291 generic.go:356] "Generic (PLEG): container finished" podID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerID="1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913" exitCode=0
Oct 30 13:12:58.549877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:58.549456    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" event={"ID":"ce3603ce-72c0-4705-9aee-66df4fd80a25","Type":"ContainerDied","Data":"1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913"}
Oct 30 13:12:58.549922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:58.549886    4291 scope.go:122] "RemoveContainer" containerID="1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913"
Oct 30 13:12:58.551236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:58.551210    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:58.551295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:58.551276    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:58.551332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:58.551306    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:12:59.577415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:12:59.577373    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" event={"ID":"ce3603ce-72c0-4705-9aee-66df4fd80a25","Type":"ContainerStarted","Data":"c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea"}
Oct 30 13:13:00.118225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:00.118191    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:13:00.388103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:00.388019    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:13:00.388272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:00.388252    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:13:00.388327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:00.388299    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:13:01.388226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:01.388189    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:13:01.388781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:01.388258    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:13:02.388319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:02.388282    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:13:02.388932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:02.388900    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:13:03.000494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.000458    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"]
Oct 30 13:13:03.000868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.000854    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" containerName="revoke-keys"
Oct 30 13:13:03.000932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.000869    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" containerName="revoke-keys"
Oct 30 13:13:03.000968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.000949    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf1ce19d-1bcc-4fa7-af3b-b961781c3b29" containerName="revoke-keys"
Oct 30 13:13:03.003459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.003431    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.031384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.031348    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"simulator-w3-5852ff-kserve-self-signed-certs\""
Oct 30 13:13:03.031609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.031496    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"openshift-service-ca.crt\""
Oct 30 13:13:03.031758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.031526    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"default-dockercfg-f7c7v\""
Oct 30 13:13:03.031758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.031692    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"kube-root-ca.crt\""
Oct 30 13:13:03.033579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.033556    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"]
Oct 30 13:13:03.076292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.076140    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-model-cache\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.076292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.076211    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-home\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.076292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.076246    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8d56\" (UniqueName: \"kubernetes.io/projected/7fc32b36-6fdb-4bfc-8d55-6240290271b9-kube-api-access-l8d56\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.076617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.076359    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-dshm\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.076617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.076432    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tmp-dir\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.076617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.076459    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tls-certs\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.182351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.182110    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-home\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.182351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.182166    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8d56\" (UniqueName: \"kubernetes.io/projected/7fc32b36-6fdb-4bfc-8d55-6240290271b9-kube-api-access-l8d56\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.182351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.182192    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-dshm\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.182351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.182241    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tmp-dir\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.182351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.182262    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tls-certs\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.182351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.182347    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-model-cache\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.182836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.182789    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-model-cache\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.183080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.183057    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-home\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.183413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.183391    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tmp-dir\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.185791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.185765    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-dshm\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.186031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.186015    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tls-certs\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.202367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.202326    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8d56\" (UniqueName: \"kubernetes.io/projected/7fc32b36-6fdb-4bfc-8d55-6240290271b9-kube-api-access-l8d56\") pod \"simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") " pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.316506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.316465    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:03.388384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.388347    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:13:03.388824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.388421    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:13:03.641653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:03.641608    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"]
Oct 30 13:13:03.643424 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:13:03.643396    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7fc32b36_6fdb_4bfc_8d55_6240290271b9.slice/crio-357741f6396a07c1f768dfa3118da7b035c7cba898dbd9332ab26b40606b9574 WatchSource:0}: Error finding container 357741f6396a07c1f768dfa3118da7b035c7cba898dbd9332ab26b40606b9574: Status 404 returned error can't find the container with id 357741f6396a07c1f768dfa3118da7b035c7cba898dbd9332ab26b40606b9574
Oct 30 13:13:04.390226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:04.390179    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Oct 30 13:13:04.390788 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:04.390248    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Oct 30 13:13:04.619827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:04.619789    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" event={"ID":"7fc32b36-6fdb-4bfc-8d55-6240290271b9","Type":"ContainerStarted","Data":"357741f6396a07c1f768dfa3118da7b035c7cba898dbd9332ab26b40606b9574"}
Oct 30 13:13:05.391700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:05.391669    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:13:05.392122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:05.392079    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:13:05.392801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:05.392778    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:13:05.627708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:05.627671    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" event={"ID":"7fc32b36-6fdb-4bfc-8d55-6240290271b9","Type":"ContainerStarted","Data":"154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02"}
Oct 30 13:13:05.667671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:05.667528    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podStartSLOduration=2.364172147 podStartE2EDuration="3.667506373s" totalImagesPullingTime="1.303334226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:05.666866105 +0000 UTC m=+595.927104339" watchObservedRunningTime="2026-10-30 13:13:05.667506373 +0000 UTC m=+595.927744606"
Oct 30 13:13:06.316565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:06.316530    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:08.210043 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:13:08.210000    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:13:09.319506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:09.319468    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:13:12.100547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.100511    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"]
Oct 30 13:13:12.105508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.105486    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.119777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.119738    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"]
Oct 30 13:13:12.129468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.129438    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"openshift-service-ca.crt\""
Oct 30 13:13:12.129623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.129489    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"kube-root-ca.crt\""
Oct 30 13:13:12.131249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.131235    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"default-dockercfg-hlqhx\""
Oct 30 13:13:12.131305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.131247    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-ec41e5\"/\"simulator-w2-ec41e5-kserve-self-signed-certs\""
Oct 30 13:13:12.174746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.174717    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-home\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.174946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.174765    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a92de03-0008-4266-b100-7ff598142395-tls-certs\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.174946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.174787    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-model-cache\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.175067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.174950    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-dshm\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.175067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.174988    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xmk4\" (UniqueName: \"kubernetes.io/projected/0a92de03-0008-4266-b100-7ff598142395-kube-api-access-6xmk4\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.175150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.175094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-tmp-dir\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276373    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-home\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276430    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a92de03-0008-4266-b100-7ff598142395-tls-certs\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276456    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-model-cache\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276490    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-dshm\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276512    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xmk4\" (UniqueName: \"kubernetes.io/projected/0a92de03-0008-4266-b100-7ff598142395-kube-api-access-6xmk4\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276607    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-tmp-dir\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276879    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-home\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.276918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276905    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-model-cache\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.277017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.276979    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-tmp-dir\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.278893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.278857    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-dshm\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.279087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.279073    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a92de03-0008-4266-b100-7ff598142395-tls-certs\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.294712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.294689    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xmk4\" (UniqueName: \"kubernetes.io/projected/0a92de03-0008-4266-b100-7ff598142395-kube-api-access-6xmk4\") pod \"simulator-w2-ec41e5-kserve-66f878ccd6-l8znh\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") " pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.419842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.419756    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:12.716818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:12.716686    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"]
Oct 30 13:13:12.720617 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:13:12.720582    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a92de03_0008_4266_b100_7ff598142395.slice/crio-f754a2e313aa41835b3e040912ecda7cb938beb93e4263b6074bd1d04221ecc0 WatchSource:0}: Error finding container f754a2e313aa41835b3e040912ecda7cb938beb93e4263b6074bd1d04221ecc0: Status 404 returned error can't find the container with id f754a2e313aa41835b3e040912ecda7cb938beb93e4263b6074bd1d04221ecc0
Oct 30 13:13:13.317308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:13.317265    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:13.330070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:13.330044    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:13.667318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:13.667287    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" event={"ID":"0a92de03-0008-4266-b100-7ff598142395","Type":"ContainerStarted","Data":"476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950"}
Oct 30 13:13:13.667512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:13.667328    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" event={"ID":"0a92de03-0008-4266-b100-7ff598142395","Type":"ContainerStarted","Data":"f754a2e313aa41835b3e040912ecda7cb938beb93e4263b6074bd1d04221ecc0"}
Oct 30 13:13:13.678687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:13.678660    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:13:13.704526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:13.704478    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podStartSLOduration=1.704463788 podStartE2EDuration="1.704463788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:13.704072469 +0000 UTC m=+603.964310681" watchObservedRunningTime="2026-10-30 13:13:13.704463788 +0000 UTC m=+603.964702022"
Oct 30 13:13:14.420291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:14.420251    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:14.924920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:14.924883    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:13:17.118966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.118925    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"]
Oct 30 13:13:17.122499 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.122472    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.151115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.151086    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"simulator-w0-f9ebb5-kserve-self-signed-certs\""
Oct 30 13:13:17.151115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.151085    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"openshift-service-ca.crt\""
Oct 30 13:13:17.151377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.151086    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"default-dockercfg-g8v59\""
Oct 30 13:13:17.151377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.151323    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"kube-root-ca.crt\""
Oct 30 13:13:17.152910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.152886    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"]
Oct 30 13:13:17.226925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.226880    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd5b4\" (UniqueName: \"kubernetes.io/projected/4b145b65-81da-48b6-afd6-5ad1ddef2572-kube-api-access-sd5b4\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.227162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.227002    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-dshm\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.227162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.227123    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b145b65-81da-48b6-afd6-5ad1ddef2572-tls-certs\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.227292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.227234    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-tmp-dir\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.227349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.227316    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-model-cache\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.227402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.227385    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-home\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328166    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd5b4\" (UniqueName: \"kubernetes.io/projected/4b145b65-81da-48b6-afd6-5ad1ddef2572-kube-api-access-sd5b4\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328212    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-dshm\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328246    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b145b65-81da-48b6-afd6-5ad1ddef2572-tls-certs\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328303    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-tmp-dir\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328321    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-model-cache\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328369    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-home\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328816    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-tmp-dir\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328840    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-home\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.328990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.328892    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-model-cache\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.330656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.330618    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-dshm\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.330981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.330961    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b145b65-81da-48b6-afd6-5ad1ddef2572-tls-certs\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.348039 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.347992    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd5b4\" (UniqueName: \"kubernetes.io/projected/4b145b65-81da-48b6-afd6-5ad1ddef2572-kube-api-access-sd5b4\") pod \"simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") " pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.437324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.437239    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:17.741721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:17.741688    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"]
Oct 30 13:13:17.744500 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:13:17.744472    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b145b65_81da_48b6_afd6_5ad1ddef2572.slice/crio-b3ef40cf69016531d421904fa820e1f883cc31658ff285340c33fbb46fa12a39 WatchSource:0}: Error finding container b3ef40cf69016531d421904fa820e1f883cc31658ff285340c33fbb46fa12a39: Status 404 returned error can't find the container with id b3ef40cf69016531d421904fa820e1f883cc31658ff285340c33fbb46fa12a39
Oct 30 13:13:18.697901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:18.697864    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" event={"ID":"4b145b65-81da-48b6-afd6-5ad1ddef2572","Type":"ContainerStarted","Data":"1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75"}
Oct 30 13:13:18.698404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:18.697912    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" event={"ID":"4b145b65-81da-48b6-afd6-5ad1ddef2572","Type":"ContainerStarted","Data":"b3ef40cf69016531d421904fa820e1f883cc31658ff285340c33fbb46fa12a39"}
Oct 30 13:13:18.737647 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:18.737582    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podStartSLOduration=1.7375657599999998 podStartE2EDuration="1.73756576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:18.735819128 +0000 UTC m=+608.996057361" watchObservedRunningTime="2026-10-30 13:13:18.73756576 +0000 UTC m=+608.997803994"
Oct 30 13:13:19.437350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:19.437309    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:22.420378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:22.420336    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:22.433049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:22.433023    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:22.730975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:22.730896    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:13:24.987944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:24.987908    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:13:27.437766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:27.437723    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:27.450704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:27.450666    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:27.755396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:27.755304    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:13:34.182809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.182774    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"]
Oct 30 13:13:34.186630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.186606    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.201371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.201339    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"kube-root-ca.crt\""
Oct 30 13:13:34.201556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.201467    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"openshift-service-ca.crt\""
Oct 30 13:13:34.201556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.201496    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"simulator-w1-52af49-kserve-self-signed-certs\""
Oct 30 13:13:34.201846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.201556    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"default-dockercfg-x88rn\""
Oct 30 13:13:34.204129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.204110    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"]
Oct 30 13:13:34.312349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.312315    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-dshm\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.312539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.312395    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-tmp-dir\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.312539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.312522    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-model-cache\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.312648 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.312568    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-home\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.312693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.312653    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f49ef7-4390-4190-badf-1b89f8505c04-tls-certs\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.312788 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.312767    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5zdw\" (UniqueName: \"kubernetes.io/projected/d2f49ef7-4390-4190-badf-1b89f8505c04-kube-api-access-b5zdw\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414228    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-model-cache\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414288    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-home\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414319    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f49ef7-4390-4190-badf-1b89f8505c04-tls-certs\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414376    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5zdw\" (UniqueName: \"kubernetes.io/projected/d2f49ef7-4390-4190-badf-1b89f8505c04-kube-api-access-b5zdw\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414441    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-dshm\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414491    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-tmp-dir\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414776    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-model-cache\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.414992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.414900    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-tmp-dir\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.415460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.415426    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-home\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.417297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.417273    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-dshm\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.417875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.417848    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f49ef7-4390-4190-badf-1b89f8505c04-tls-certs\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.433575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.433508    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5zdw\" (UniqueName: \"kubernetes.io/projected/d2f49ef7-4390-4190-badf-1b89f8505c04-kube-api-access-b5zdw\") pod \"simulator-w1-52af49-kserve-64b7746d65-q65rl\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") " pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.501451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.501418    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:34.805445 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:34.805422    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"]
Oct 30 13:13:34.807484 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:13:34.807454    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd2f49ef7_4390_4190_badf_1b89f8505c04.slice/crio-ed9a7e1bf59867436f62a54bfaef7633909d823fff541fdb9da3366a81e6c768 WatchSource:0}: Error finding container ed9a7e1bf59867436f62a54bfaef7633909d823fff541fdb9da3366a81e6c768: Status 404 returned error can't find the container with id ed9a7e1bf59867436f62a54bfaef7633909d823fff541fdb9da3366a81e6c768
Oct 30 13:13:35.785980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:35.785947    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" event={"ID":"d2f49ef7-4390-4190-badf-1b89f8505c04","Type":"ContainerStarted","Data":"9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23"}
Oct 30 13:13:35.786389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:35.785988    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" event={"ID":"d2f49ef7-4390-4190-badf-1b89f8505c04","Type":"ContainerStarted","Data":"ed9a7e1bf59867436f62a54bfaef7633909d823fff541fdb9da3366a81e6c768"}
Oct 30 13:13:35.823482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:35.823434    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podStartSLOduration=1.823420516 podStartE2EDuration="1.823420516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:35.823162048 +0000 UTC m=+626.083400288" watchObservedRunningTime="2026-10-30 13:13:35.823420516 +0000 UTC m=+626.083658749"
Oct 30 13:13:36.502042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:36.502013    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:37.233058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:37.233028    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:13:38.209827 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:13:38.209782    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:13:44.501705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:44.501667    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:44.518396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:44.518363    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:44.842691 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:44.842587    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:13:48.827586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:48.827549    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:13:53.469526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.469488    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"]
Oct 30 13:13:53.476714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.474802    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.490794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.490768    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tmp-dir\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.490955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.490824    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tls-certs\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.491032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.490948    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-model-cache\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.491032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.490988    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-dshm\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.491136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.491062    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-home\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.491136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.491100    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd5zk\" (UniqueName: \"kubernetes.io/projected/78fe0cce-0dda-410b-bf07-5c5db90c52b8-kube-api-access-fd5zk\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.501450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.501424    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"premium-w0-f9ebb5-kserve-self-signed-certs\""
Oct 30 13:13:53.503356 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.503318    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"]
Oct 30 13:13:53.592517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.592475    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tls-certs\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.592712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.592565    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-model-cache\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.592712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.592596    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-dshm\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.592712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.592654    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-home\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.592712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.592699    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd5zk\" (UniqueName: \"kubernetes.io/projected/78fe0cce-0dda-410b-bf07-5c5db90c52b8-kube-api-access-fd5zk\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.592939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.592748    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tmp-dir\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.593402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.593377    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-model-cache\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.593511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.593373    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-home\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.593511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.593454    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tmp-dir\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.595493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.595461    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-dshm\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.595740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.595721    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tls-certs\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.612541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.612502    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd5zk\" (UniqueName: \"kubernetes.io/projected/78fe0cce-0dda-410b-bf07-5c5db90c52b8-kube-api-access-fd5zk\") pod \"premium-w0-f9ebb5-kserve-84997cf747-f76wd\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") " pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:53.788158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:53.788083    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:13:54.096963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:54.096867    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"]
Oct 30 13:13:54.099311 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:13:54.099287    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod78fe0cce_0dda_410b_bf07_5c5db90c52b8.slice/crio-7b2526d081f2cd61b312b151900ec65be26e5927a252fd56683b231b5e0052fb WatchSource:0}: Error finding container 7b2526d081f2cd61b312b151900ec65be26e5927a252fd56683b231b5e0052fb: Status 404 returned error can't find the container with id 7b2526d081f2cd61b312b151900ec65be26e5927a252fd56683b231b5e0052fb
Oct 30 13:13:54.883087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:54.883051    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" event={"ID":"78fe0cce-0dda-410b-bf07-5c5db90c52b8","Type":"ContainerStarted","Data":"53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994"}
Oct 30 13:13:54.883087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:54.883092    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" event={"ID":"78fe0cce-0dda-410b-bf07-5c5db90c52b8","Type":"ContainerStarted","Data":"7b2526d081f2cd61b312b151900ec65be26e5927a252fd56683b231b5e0052fb"}
Oct 30 13:13:54.920846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:54.920784    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podStartSLOduration=1.920766885 podStartE2EDuration="1.920766885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:13:54.920136058 +0000 UTC m=+645.180374294" watchObservedRunningTime="2026-10-30 13:13:54.920766885 +0000 UTC m=+645.181005121"
Oct 30 13:13:55.788693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:13:55.788663    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:14:03.789230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:03.789198    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:14:03.802344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:03.802320    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:14:03.960366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:03.960329    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:14:06.815511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:06.815473    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:14:08.209390 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:14:08.209353    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:14:17.386278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.386233    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"]
Oct 30 13:14:17.391072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.391052    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.408326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.407081    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"]
Oct 30 13:14:17.416451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.416420    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:14:17.416597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.416536    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:14:17.416597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.416577    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:14:17.417298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.417281    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:14:17.530125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.530074    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkczp\" (UniqueName: \"kubernetes.io/projected/f19001ae-04a7-431a-8d1c-06ec4d981a71-kube-api-access-lkczp\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.530344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.530146    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.530413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.530329    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.631458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.631416    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.631664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.631507    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkczp\" (UniqueName: \"kubernetes.io/projected/f19001ae-04a7-431a-8d1c-06ec4d981a71-kube-api-access-lkczp\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.631664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.631550    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.632280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.632256    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.633886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.633862    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.650135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.650079    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkczp\" (UniqueName: \"kubernetes.io/projected/f19001ae-04a7-431a-8d1c-06ec4d981a71-kube-api-access-lkczp\") pod \"maas-api-revoke-sub-d2a0c5b8-x4j87\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:17.706472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:17.706438    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:18.008010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:18.007987    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"]
Oct 30 13:14:18.010147 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:14:18.010121    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf19001ae_04a7_431a_8d1c_06ec4d981a71.slice/crio-acaff2f7f5bf34535f1f58e4889b5b1793be22015c0317520add2916fbb7accc WatchSource:0}: Error finding container acaff2f7f5bf34535f1f58e4889b5b1793be22015c0317520add2916fbb7accc: Status 404 returned error can't find the container with id acaff2f7f5bf34535f1f58e4889b5b1793be22015c0317520add2916fbb7accc
Oct 30 13:14:18.022102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:18.022078    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87" event={"ID":"f19001ae-04a7-431a-8d1c-06ec4d981a71","Type":"ContainerStarted","Data":"acaff2f7f5bf34535f1f58e4889b5b1793be22015c0317520add2916fbb7accc"}
Oct 30 13:14:19.034029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:19.033931    4291 generic.go:356] "Generic (PLEG): container finished" podID="f19001ae-04a7-431a-8d1c-06ec4d981a71" containerID="a436e5961f005cec9e3ea63464037cdf14217f448c4adaaca9815244a1df8e74" exitCode=0
Oct 30 13:14:19.034029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:19.033999    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87" event={"ID":"f19001ae-04a7-431a-8d1c-06ec4d981a71","Type":"ContainerDied","Data":"a436e5961f005cec9e3ea63464037cdf14217f448c4adaaca9815244a1df8e74"}
Oct 30 13:14:20.178770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.178741    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:20.259199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.259162    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-openshift-ca\") pod \"f19001ae-04a7-431a-8d1c-06ec4d981a71\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") "
Oct 30 13:14:20.259199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.259204    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-xks-ca\") pod \"f19001ae-04a7-431a-8d1c-06ec4d981a71\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") "
Oct 30 13:14:20.259454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.259332    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f19001ae-04a7-431a-8d1c-06ec4d981a71-kube-api-access-lkczp\" (UniqueName: \"kubernetes.io/projected/f19001ae-04a7-431a-8d1c-06ec4d981a71-kube-api-access-lkczp\") pod \"f19001ae-04a7-431a-8d1c-06ec4d981a71\" (UID: \"f19001ae-04a7-431a-8d1c-06ec4d981a71\") "
Oct 30 13:14:20.259615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.259590    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-openshift-ca" pod "f19001ae-04a7-431a-8d1c-06ec4d981a71" (UID: "f19001ae-04a7-431a-8d1c-06ec4d981a71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:20.261361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.261341    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f19001ae-04a7-431a-8d1c-06ec4d981a71-kube-api-access-lkczp" pod "f19001ae-04a7-431a-8d1c-06ec4d981a71" (UID: "f19001ae-04a7-431a-8d1c-06ec4d981a71"). InnerVolumeSpecName "kube-api-access-lkczp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:20.261441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.261410    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-xks-ca" pod "f19001ae-04a7-431a-8d1c-06ec4d981a71" (UID: "f19001ae-04a7-431a-8d1c-06ec4d981a71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:20.360251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.360220    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkczp\" (UniqueName: \"kubernetes.io/projected/f19001ae-04a7-431a-8d1c-06ec4d981a71-kube-api-access-lkczp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:20.360251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.360248    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:20.360443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:20.360260    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f19001ae-04a7-431a-8d1c-06ec4d981a71-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:21.045826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:21.045794    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"
Oct 30 13:14:21.045826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:21.045803    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87" event={"ID":"f19001ae-04a7-431a-8d1c-06ec4d981a71","Type":"ContainerDied","Data":"acaff2f7f5bf34535f1f58e4889b5b1793be22015c0317520add2916fbb7accc"}
Oct 30 13:14:21.046033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:21.045846    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="acaff2f7f5bf34535f1f58e4889b5b1793be22015c0317520add2916fbb7accc"
Oct 30 13:14:28.199211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:28.199138    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:14:32.927135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.927102    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"]
Oct 30 13:14:32.927515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.927468    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f19001ae-04a7-431a-8d1c-06ec4d981a71" containerName="revoke-keys"
Oct 30 13:14:32.927515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.927479    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f19001ae-04a7-431a-8d1c-06ec4d981a71" containerName="revoke-keys"
Oct 30 13:14:32.927595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.927548    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f19001ae-04a7-431a-8d1c-06ec4d981a71" containerName="revoke-keys"
Oct 30 13:14:32.930738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.930722    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:32.952046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.952021    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"]
Oct 30 13:14:32.961391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.961362    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:14:32.961391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.961376    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:14:32.961391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.961389    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:14:32.961593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.961389    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:14:32.974902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.974872    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrk5v\" (UniqueName: \"kubernetes.io/projected/43901a56-9404-4f14-9962-9540261123f9-kube-api-access-jrk5v\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:32.975016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.974909    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901a56-9404-4f14-9962-9540261123f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:32.975118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:32.975094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901a56-9404-4f14-9962-9540261123f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.075763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.075732    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901a56-9404-4f14-9962-9540261123f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.075947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.075805    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrk5v\" (UniqueName: \"kubernetes.io/projected/43901a56-9404-4f14-9962-9540261123f9-kube-api-access-jrk5v\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.075947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.075829    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901a56-9404-4f14-9962-9540261123f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.076451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.076429    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901a56-9404-4f14-9962-9540261123f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.078095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.078077    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901a56-9404-4f14-9962-9540261123f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.106759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.106732    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrk5v\" (UniqueName: \"kubernetes.io/projected/43901a56-9404-4f14-9962-9540261123f9-kube-api-access-jrk5v\") pod \"maas-api-revoke-sub-2cccafb7-vvk5x\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.244107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.244022    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:33.546478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:33.546448    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"]
Oct 30 13:14:33.548686 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:14:33.548661    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43901a56_9404_4f14_9962_9540261123f9.slice/crio-6699316070d30def110cbf73cc62e4b257d86c0b01020a93eaaac0b8bd2c14c6 WatchSource:0}: Error finding container 6699316070d30def110cbf73cc62e4b257d86c0b01020a93eaaac0b8bd2c14c6: Status 404 returned error can't find the container with id 6699316070d30def110cbf73cc62e4b257d86c0b01020a93eaaac0b8bd2c14c6
Oct 30 13:14:34.108842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:34.108755    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x" event={"ID":"43901a56-9404-4f14-9962-9540261123f9","Type":"ContainerStarted","Data":"192b0485fa8e232beb227c89feda6a751050b8f4cace74fa9f03d56cabce9e7a"}
Oct 30 13:14:34.108842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:34.108818    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x" event={"ID":"43901a56-9404-4f14-9962-9540261123f9","Type":"ContainerStarted","Data":"6699316070d30def110cbf73cc62e4b257d86c0b01020a93eaaac0b8bd2c14c6"}
Oct 30 13:14:34.151055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:34.151004    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x" podStartSLOduration=2.150990101 podStartE2EDuration="2.150990101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:34.150308144 +0000 UTC m=+684.410546379" watchObservedRunningTime="2026-10-30 13:14:34.150990101 +0000 UTC m=+684.411228334"
Oct 30 13:14:35.114703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:35.114667    4291 generic.go:356] "Generic (PLEG): container finished" podID="43901a56-9404-4f14-9962-9540261123f9" containerID="192b0485fa8e232beb227c89feda6a751050b8f4cace74fa9f03d56cabce9e7a" exitCode=0
Oct 30 13:14:35.115090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:35.114743    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x" event={"ID":"43901a56-9404-4f14-9962-9540261123f9","Type":"ContainerDied","Data":"192b0485fa8e232beb227c89feda6a751050b8f4cace74fa9f03d56cabce9e7a"}
Oct 30 13:14:36.254346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.254317    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:36.304944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.304913    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43901a56-9404-4f14-9962-9540261123f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901a56-9404-4f14-9962-9540261123f9-maas-api-xks-ca\") pod \"43901a56-9404-4f14-9962-9540261123f9\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") "
Oct 30 13:14:36.305082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.305010    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43901a56-9404-4f14-9962-9540261123f9-kube-api-access-jrk5v\" (UniqueName: \"kubernetes.io/projected/43901a56-9404-4f14-9962-9540261123f9-kube-api-access-jrk5v\") pod \"43901a56-9404-4f14-9962-9540261123f9\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") "
Oct 30 13:14:36.305129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.305078    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43901a56-9404-4f14-9962-9540261123f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901a56-9404-4f14-9962-9540261123f9-maas-api-openshift-ca\") pod \"43901a56-9404-4f14-9962-9540261123f9\" (UID: \"43901a56-9404-4f14-9962-9540261123f9\") "
Oct 30 13:14:36.305453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.305422    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43901a56-9404-4f14-9962-9540261123f9-maas-api-openshift-ca" pod "43901a56-9404-4f14-9962-9540261123f9" (UID: "43901a56-9404-4f14-9962-9540261123f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:36.306984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.306964    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43901a56-9404-4f14-9962-9540261123f9-maas-api-xks-ca" pod "43901a56-9404-4f14-9962-9540261123f9" (UID: "43901a56-9404-4f14-9962-9540261123f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:36.307064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.307028    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43901a56-9404-4f14-9962-9540261123f9-kube-api-access-jrk5v" pod "43901a56-9404-4f14-9962-9540261123f9" (UID: "43901a56-9404-4f14-9962-9540261123f9"). InnerVolumeSpecName "kube-api-access-jrk5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:36.406966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.406890    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43901a56-9404-4f14-9962-9540261123f9-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:36.406966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.406918    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrk5v\" (UniqueName: \"kubernetes.io/projected/43901a56-9404-4f14-9962-9540261123f9-kube-api-access-jrk5v\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:36.406966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:36.406933    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43901a56-9404-4f14-9962-9540261123f9-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:37.125658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:37.125605    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"
Oct 30 13:14:37.125838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:37.125671    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x" event={"ID":"43901a56-9404-4f14-9962-9540261123f9","Type":"ContainerDied","Data":"6699316070d30def110cbf73cc62e4b257d86c0b01020a93eaaac0b8bd2c14c6"}
Oct 30 13:14:37.125838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:37.125728    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6699316070d30def110cbf73cc62e4b257d86c0b01020a93eaaac0b8bd2c14c6"
Oct 30 13:14:38.209093 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:14:38.209062    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:14:41.628386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.628348    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"]
Oct 30 13:14:41.628804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.628751    4291 cpu_manager.go:414] "containerMap: removing container" podUID="43901a56-9404-4f14-9962-9540261123f9" containerName="revoke-keys"
Oct 30 13:14:41.628804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.628762    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43901a56-9404-4f14-9962-9540261123f9" containerName="revoke-keys"
Oct 30 13:14:41.628881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.628831    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="43901a56-9404-4f14-9962-9540261123f9" containerName="revoke-keys"
Oct 30 13:14:41.632062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.632039    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.651082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.651051    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:14:41.651358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.651343    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:14:41.663329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.663311    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:14:41.663329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.663311    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:14:41.688357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.688333    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"]
Oct 30 13:14:41.756146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.756118    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.756304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.756166    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.756347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.756300    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78bmr\" (UniqueName: \"kubernetes.io/projected/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-kube-api-access-78bmr\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.857871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.857833    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78bmr\" (UniqueName: \"kubernetes.io/projected/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-kube-api-access-78bmr\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.858042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.857909    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.858042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.857976    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.858547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.858524    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.860255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.860233    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.879680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.879602    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78bmr\" (UniqueName: \"kubernetes.io/projected/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-kube-api-access-78bmr\") pod \"maas-api-revoke-sub-f3731ddd-5lpp6\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:41.945267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:41.945231    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:42.248245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:42.248170    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"]
Oct 30 13:14:42.254459 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:14:42.254431    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f4c9845_91fe_49ec_be3d_6f04d0fbc864.slice/crio-dc48a98d4853a58d21a24969b70f53c590046a63c3e8dd221f931b774a2094e3 WatchSource:0}: Error finding container dc48a98d4853a58d21a24969b70f53c590046a63c3e8dd221f931b774a2094e3: Status 404 returned error can't find the container with id dc48a98d4853a58d21a24969b70f53c590046a63c3e8dd221f931b774a2094e3
Oct 30 13:14:43.156055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:43.156023    4291 generic.go:356] "Generic (PLEG): container finished" podID="7f4c9845-91fe-49ec-be3d-6f04d0fbc864" containerID="acc185fecb5b93b7b5553934f8211f9b9cf84afd38bd836a5c861f73696d2fd6" exitCode=0
Oct 30 13:14:43.156409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:43.156105    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6" event={"ID":"7f4c9845-91fe-49ec-be3d-6f04d0fbc864","Type":"ContainerDied","Data":"acc185fecb5b93b7b5553934f8211f9b9cf84afd38bd836a5c861f73696d2fd6"}
Oct 30 13:14:43.156409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:43.156158    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6" event={"ID":"7f4c9845-91fe-49ec-be3d-6f04d0fbc864","Type":"ContainerStarted","Data":"dc48a98d4853a58d21a24969b70f53c590046a63c3e8dd221f931b774a2094e3"}
Oct 30 13:14:44.312403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.312373    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:44.387605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.387579    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-kube-api-access-78bmr\" (UniqueName: \"kubernetes.io/projected/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-kube-api-access-78bmr\") pod \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") "
Oct 30 13:14:44.387829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.387644    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-xks-ca\") pod \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") "
Oct 30 13:14:44.387829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.387681    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-openshift-ca\") pod \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\" (UID: \"7f4c9845-91fe-49ec-be3d-6f04d0fbc864\") "
Oct 30 13:14:44.388131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.388106    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-openshift-ca" pod "7f4c9845-91fe-49ec-be3d-6f04d0fbc864" (UID: "7f4c9845-91fe-49ec-be3d-6f04d0fbc864"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:44.389781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.389752    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-kube-api-access-78bmr" pod "7f4c9845-91fe-49ec-be3d-6f04d0fbc864" (UID: "7f4c9845-91fe-49ec-be3d-6f04d0fbc864"). InnerVolumeSpecName "kube-api-access-78bmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:44.389781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.389763    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-xks-ca" pod "7f4c9845-91fe-49ec-be3d-6f04d0fbc864" (UID: "7f4c9845-91fe-49ec-be3d-6f04d0fbc864"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:44.488784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.488705    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:44.488784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.488731    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78bmr\" (UniqueName: \"kubernetes.io/projected/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-kube-api-access-78bmr\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:44.488784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:44.488741    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f4c9845-91fe-49ec-be3d-6f04d0fbc864-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:45.167068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:45.167040    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"
Oct 30 13:14:45.167237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:45.167079    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6" event={"ID":"7f4c9845-91fe-49ec-be3d-6f04d0fbc864","Type":"ContainerDied","Data":"dc48a98d4853a58d21a24969b70f53c590046a63c3e8dd221f931b774a2094e3"}
Oct 30 13:14:45.167237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:45.167122    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc48a98d4853a58d21a24969b70f53c590046a63c3e8dd221f931b774a2094e3"
Oct 30 13:14:47.631434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.631401    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:14:47.683349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.683321    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"]
Oct 30 13:14:47.683724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.683711    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7f4c9845-91fe-49ec-be3d-6f04d0fbc864" containerName="revoke-keys"
Oct 30 13:14:47.683776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.683725    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f4c9845-91fe-49ec-be3d-6f04d0fbc864" containerName="revoke-keys"
Oct 30 13:14:47.683811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.683799    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f4c9845-91fe-49ec-be3d-6f04d0fbc864" containerName="revoke-keys"
Oct 30 13:14:47.686873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.686855    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.701741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.701720    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:14:47.713414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.713392    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:14:47.713414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.713391    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:14:47.713546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.713513    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:14:47.715404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.715386    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"]
Oct 30 13:14:47.818126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.818091    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.818275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.818155    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.818275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.818265    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkmfd\" (UniqueName: \"kubernetes.io/projected/504c5f3c-8117-4a37-9de3-a198f3f48812-kube-api-access-kkmfd\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.919151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.919068    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkmfd\" (UniqueName: \"kubernetes.io/projected/504c5f3c-8117-4a37-9de3-a198f3f48812-kube-api-access-kkmfd\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.919268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.919153    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.919268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.919201    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.919813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.919796    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.921323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.921306    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.942304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.942276    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkmfd\" (UniqueName: \"kubernetes.io/projected/504c5f3c-8117-4a37-9de3-a198f3f48812-kube-api-access-kkmfd\") pod \"maas-api-revoke-sub-aa7bac55-vjp74\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:47.999383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:47.999357    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:48.299498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:48.299470    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"]
Oct 30 13:14:48.301866 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:14:48.301831    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod504c5f3c_8117_4a37_9de3_a198f3f48812.slice/crio-e4dee1b89a74d5266de6f71addfb5782d2bb6fa0956a31040be691c08e030f84 WatchSource:0}: Error finding container e4dee1b89a74d5266de6f71addfb5782d2bb6fa0956a31040be691c08e030f84: Status 404 returned error can't find the container with id e4dee1b89a74d5266de6f71addfb5782d2bb6fa0956a31040be691c08e030f84
Oct 30 13:14:49.188852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:49.188811    4291 generic.go:356] "Generic (PLEG): container finished" podID="504c5f3c-8117-4a37-9de3-a198f3f48812" containerID="fd5ca7ef7c1dec145f62f21a81a9915083ebfc88610e7d1659e419d1219f8b4a" exitCode=0
Oct 30 13:14:49.189267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:49.188893    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74" event={"ID":"504c5f3c-8117-4a37-9de3-a198f3f48812","Type":"ContainerDied","Data":"fd5ca7ef7c1dec145f62f21a81a9915083ebfc88610e7d1659e419d1219f8b4a"}
Oct 30 13:14:49.189267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:49.188976    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74" event={"ID":"504c5f3c-8117-4a37-9de3-a198f3f48812","Type":"ContainerStarted","Data":"e4dee1b89a74d5266de6f71addfb5782d2bb6fa0956a31040be691c08e030f84"}
Oct 30 13:14:50.326972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.326947    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:50.441743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.441715    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/504c5f3c-8117-4a37-9de3-a198f3f48812-kube-api-access-kkmfd\" (UniqueName: \"kubernetes.io/projected/504c5f3c-8117-4a37-9de3-a198f3f48812-kube-api-access-kkmfd\") pod \"504c5f3c-8117-4a37-9de3-a198f3f48812\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") "
Oct 30 13:14:50.441902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.441773    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-xks-ca\") pod \"504c5f3c-8117-4a37-9de3-a198f3f48812\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") "
Oct 30 13:14:50.441902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.441846    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-openshift-ca\") pod \"504c5f3c-8117-4a37-9de3-a198f3f48812\" (UID: \"504c5f3c-8117-4a37-9de3-a198f3f48812\") "
Oct 30 13:14:50.442270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.442232    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-openshift-ca" pod "504c5f3c-8117-4a37-9de3-a198f3f48812" (UID: "504c5f3c-8117-4a37-9de3-a198f3f48812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:50.443818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.443792    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/504c5f3c-8117-4a37-9de3-a198f3f48812-kube-api-access-kkmfd" pod "504c5f3c-8117-4a37-9de3-a198f3f48812" (UID: "504c5f3c-8117-4a37-9de3-a198f3f48812"). InnerVolumeSpecName "kube-api-access-kkmfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:50.444023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.443999    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-xks-ca" pod "504c5f3c-8117-4a37-9de3-a198f3f48812" (UID: "504c5f3c-8117-4a37-9de3-a198f3f48812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:50.542943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.542909    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkmfd\" (UniqueName: \"kubernetes.io/projected/504c5f3c-8117-4a37-9de3-a198f3f48812-kube-api-access-kkmfd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:50.542943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.542938    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:50.542943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:50.542948    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/504c5f3c-8117-4a37-9de3-a198f3f48812-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:51.199781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:51.199749    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"
Oct 30 13:14:51.199781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:51.199760    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74" event={"ID":"504c5f3c-8117-4a37-9de3-a198f3f48812","Type":"ContainerDied","Data":"e4dee1b89a74d5266de6f71addfb5782d2bb6fa0956a31040be691c08e030f84"}
Oct 30 13:14:51.200069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:51.199809    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4dee1b89a74d5266de6f71addfb5782d2bb6fa0956a31040be691c08e030f84"
Oct 30 13:14:55.932780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.932735    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"]
Oct 30 13:14:55.933174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.933151    4291 cpu_manager.go:414] "containerMap: removing container" podUID="504c5f3c-8117-4a37-9de3-a198f3f48812" containerName="revoke-keys"
Oct 30 13:14:55.933174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.933163    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="504c5f3c-8117-4a37-9de3-a198f3f48812" containerName="revoke-keys"
Oct 30 13:14:55.933252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.933234    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="504c5f3c-8117-4a37-9de3-a198f3f48812" containerName="revoke-keys"
Oct 30 13:14:55.936223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.936205    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:55.951252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.951220    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:14:55.951378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.951262    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:14:55.952194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.952173    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"]
Oct 30 13:14:55.961451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.961432    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:14:55.961451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:55.961436    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:14:56.097565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.097529    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82rjk\" (UniqueName: \"kubernetes.io/projected/f725e253-a9ce-43ab-88ed-3a373c32af71-kube-api-access-82rjk\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.097752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.097678    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.097752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.097744    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.199221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.199140    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.199221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.199189    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.199419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.199302    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82rjk\" (UniqueName: \"kubernetes.io/projected/f725e253-a9ce-43ab-88ed-3a373c32af71-kube-api-access-82rjk\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.199867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.199846    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.201520 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.201504    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.218934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.218906    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82rjk\" (UniqueName: \"kubernetes.io/projected/f725e253-a9ce-43ab-88ed-3a373c32af71-kube-api-access-82rjk\") pod \"maas-api-revoke-sub-23ef410d-mz2nn\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.248792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.248764    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:56.548020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:56.547995    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"]
Oct 30 13:14:56.550376 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:14:56.550347    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf725e253_a9ce_43ab_88ed_3a373c32af71.slice/crio-efd54a157395a5d0185f92c73a2bd1b03738d6586f9b9213513a7dc515c741af WatchSource:0}: Error finding container efd54a157395a5d0185f92c73a2bd1b03738d6586f9b9213513a7dc515c741af: Status 404 returned error can't find the container with id efd54a157395a5d0185f92c73a2bd1b03738d6586f9b9213513a7dc515c741af
Oct 30 13:14:57.236019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:57.235939    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn" event={"ID":"f725e253-a9ce-43ab-88ed-3a373c32af71","Type":"ContainerStarted","Data":"ce78ab0bf40ecd3224fe194f721cdd5e02c1fdb6de38cd002b2e26e539891cf5"}
Oct 30 13:14:57.236019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:57.235993    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn" event={"ID":"f725e253-a9ce-43ab-88ed-3a373c32af71","Type":"ContainerStarted","Data":"efd54a157395a5d0185f92c73a2bd1b03738d6586f9b9213513a7dc515c741af"}
Oct 30 13:14:57.273440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:57.273385    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn" podStartSLOduration=2.273371375 podStartE2EDuration="2.273371375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:57.273082434 +0000 UTC m=+707.533320659" watchObservedRunningTime="2026-10-30 13:14:57.273371375 +0000 UTC m=+707.533609609"
Oct 30 13:14:58.044459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.044427    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"]
Oct 30 13:14:58.047851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.047834    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.064284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.064255    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"]
Oct 30 13:14:58.120969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.120932    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.121150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.120990    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.121150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.121121    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-87mjq\" (UniqueName: \"kubernetes.io/projected/972f6718-00ca-47e1-9ccb-c52d884ed69f-kube-api-access-87mjq\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.221707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.221660    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-87mjq\" (UniqueName: \"kubernetes.io/projected/972f6718-00ca-47e1-9ccb-c52d884ed69f-kube-api-access-87mjq\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.221930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.221797    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.221930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.221844    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.222433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.222402    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.224234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.224214    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.241186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.241156    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-87mjq\" (UniqueName: \"kubernetes.io/projected/972f6718-00ca-47e1-9ccb-c52d884ed69f-kube-api-access-87mjq\") pod \"maas-api-revoke-sub-eae4bc3f-k772s\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.242205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.242182    4291 generic.go:356] "Generic (PLEG): container finished" podID="f725e253-a9ce-43ab-88ed-3a373c32af71" containerID="ce78ab0bf40ecd3224fe194f721cdd5e02c1fdb6de38cd002b2e26e539891cf5" exitCode=0
Oct 30 13:14:58.242295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.242275    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn" event={"ID":"f725e253-a9ce-43ab-88ed-3a373c32af71","Type":"ContainerDied","Data":"ce78ab0bf40ecd3224fe194f721cdd5e02c1fdb6de38cd002b2e26e539891cf5"}
Oct 30 13:14:58.360271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.360241    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:14:58.665801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:58.665773    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"]
Oct 30 13:14:58.667006 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:14:58.666982    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972f6718_00ca_47e1_9ccb_c52d884ed69f.slice/crio-20064fab619ae8b631cfd4e3863ea5968faf0c09648d3702c96d911bc9373627 WatchSource:0}: Error finding container 20064fab619ae8b631cfd4e3863ea5968faf0c09648d3702c96d911bc9373627: Status 404 returned error can't find the container with id 20064fab619ae8b631cfd4e3863ea5968faf0c09648d3702c96d911bc9373627
Oct 30 13:14:59.249322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.249233    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s" event={"ID":"972f6718-00ca-47e1-9ccb-c52d884ed69f","Type":"ContainerStarted","Data":"4e6dcad551c70b92ac7e28c7245cea2d69ee5dd34cfe85ab3258351f3ff956f8"}
Oct 30 13:14:59.249780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.249328    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s" event={"ID":"972f6718-00ca-47e1-9ccb-c52d884ed69f","Type":"ContainerStarted","Data":"20064fab619ae8b631cfd4e3863ea5968faf0c09648d3702c96d911bc9373627"}
Oct 30 13:14:59.289428 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.289369    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s" podStartSLOduration=1.289353757 podStartE2EDuration="1.289353757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:14:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:14:59.287277172 +0000 UTC m=+709.547515409" watchObservedRunningTime="2026-10-30 13:14:59.289353757 +0000 UTC m=+709.549591991"
Oct 30 13:14:59.387653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.387617    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:14:59.534917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.534824    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f725e253-a9ce-43ab-88ed-3a373c32af71-kube-api-access-82rjk\" (UniqueName: \"kubernetes.io/projected/f725e253-a9ce-43ab-88ed-3a373c32af71-kube-api-access-82rjk\") pod \"f725e253-a9ce-43ab-88ed-3a373c32af71\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") "
Oct 30 13:14:59.534917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.534870    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-openshift-ca\") pod \"f725e253-a9ce-43ab-88ed-3a373c32af71\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") "
Oct 30 13:14:59.535154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.534958    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-xks-ca\") pod \"f725e253-a9ce-43ab-88ed-3a373c32af71\" (UID: \"f725e253-a9ce-43ab-88ed-3a373c32af71\") "
Oct 30 13:14:59.535345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.535320    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-openshift-ca" pod "f725e253-a9ce-43ab-88ed-3a373c32af71" (UID: "f725e253-a9ce-43ab-88ed-3a373c32af71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:14:59.550049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.550015    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-xks-ca" pod "f725e253-a9ce-43ab-88ed-3a373c32af71" (UID: "f725e253-a9ce-43ab-88ed-3a373c32af71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:14:59.550049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.550020    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f725e253-a9ce-43ab-88ed-3a373c32af71-kube-api-access-82rjk" pod "f725e253-a9ce-43ab-88ed-3a373c32af71" (UID: "f725e253-a9ce-43ab-88ed-3a373c32af71"). InnerVolumeSpecName "kube-api-access-82rjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:14:59.636613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.636578    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82rjk\" (UniqueName: \"kubernetes.io/projected/f725e253-a9ce-43ab-88ed-3a373c32af71-kube-api-access-82rjk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.636613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.636608    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:14:59.636613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:14:59.636618    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f725e253-a9ce-43ab-88ed-3a373c32af71-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:00.190306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.190221    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"]
Oct 30 13:15:00.190620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.190608    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f725e253-a9ce-43ab-88ed-3a373c32af71" containerName="revoke-keys"
Oct 30 13:15:00.190678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.190621    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f725e253-a9ce-43ab-88ed-3a373c32af71" containerName="revoke-keys"
Oct 30 13:15:00.190747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.190737    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f725e253-a9ce-43ab-88ed-3a373c32af71" containerName="revoke-keys"
Oct 30 13:15:00.193769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.193751    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.202381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.202354    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"]
Oct 30 13:15:00.205804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.205781    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"]
Oct 30 13:15:00.205937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.205904    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.209195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.209171    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.218462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.217457    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"]
Oct 30 13:15:00.235733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.235584    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"]
Oct 30 13:15:00.238487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.238464    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"]
Oct 30 13:15:00.256374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.256344    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"
Oct 30 13:15:00.256847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.256377    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn" event={"ID":"f725e253-a9ce-43ab-88ed-3a373c32af71","Type":"ContainerDied","Data":"efd54a157395a5d0185f92c73a2bd1b03738d6586f9b9213513a7dc515c741af"}
Oct 30 13:15:00.256847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.256430    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="efd54a157395a5d0185f92c73a2bd1b03738d6586f9b9213513a7dc515c741af"
Oct 30 13:15:00.258540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.258516    4291 generic.go:356] "Generic (PLEG): container finished" podID="972f6718-00ca-47e1-9ccb-c52d884ed69f" containerID="4e6dcad551c70b92ac7e28c7245cea2d69ee5dd34cfe85ab3258351f3ff956f8" exitCode=0
Oct 30 13:15:00.258710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.258627    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s" event={"ID":"972f6718-00ca-47e1-9ccb-c52d884ed69f","Type":"ContainerDied","Data":"4e6dcad551c70b92ac7e28c7245cea2d69ee5dd34cfe85ab3258351f3ff956f8"}
Oct 30 13:15:00.342803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.342770    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztcxr\" (UniqueName: \"kubernetes.io/projected/c929656c-5225-4483-a0f2-aff776e94a5b-kube-api-access-ztcxr\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.342967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.342805    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxqvg\" (UniqueName: \"kubernetes.io/projected/4cd508fb-6df4-480e-8a39-1acbeb90d183-kube-api-access-qxqvg\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.342967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.342916    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c929656c-5225-4483-a0f2-aff776e94a5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.343060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.343009    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f924632e-8a3c-48d2-aa7e-74096322bc7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.343060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.343047    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hl5\" (UniqueName: \"kubernetes.io/projected/f924632e-8a3c-48d2-aa7e-74096322bc7e-kube-api-access-d4hl5\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.343123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.343084    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd508fb-6df4-480e-8a39-1acbeb90d183-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.444409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.444318    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztcxr\" (UniqueName: \"kubernetes.io/projected/c929656c-5225-4483-a0f2-aff776e94a5b-kube-api-access-ztcxr\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.444409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.444354    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxqvg\" (UniqueName: \"kubernetes.io/projected/4cd508fb-6df4-480e-8a39-1acbeb90d183-kube-api-access-qxqvg\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.444409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.444374    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c929656c-5225-4483-a0f2-aff776e94a5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.444711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.444435    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f924632e-8a3c-48d2-aa7e-74096322bc7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.444711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.444451    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hl5\" (UniqueName: \"kubernetes.io/projected/f924632e-8a3c-48d2-aa7e-74096322bc7e-kube-api-access-d4hl5\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.444711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.444572    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd508fb-6df4-480e-8a39-1acbeb90d183-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.445188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.445168    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f924632e-8a3c-48d2-aa7e-74096322bc7e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.445249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.445176    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c929656c-5225-4483-a0f2-aff776e94a5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.445306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.445259    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd508fb-6df4-480e-8a39-1acbeb90d183-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.469697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.469629    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hl5\" (UniqueName: \"kubernetes.io/projected/f924632e-8a3c-48d2-aa7e-74096322bc7e-kube-api-access-d4hl5\") pod \"maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.469844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.469695    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxqvg\" (UniqueName: \"kubernetes.io/projected/4cd508fb-6df4-480e-8a39-1acbeb90d183-kube-api-access-qxqvg\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.470257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.470241    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztcxr\" (UniqueName: \"kubernetes.io/projected/c929656c-5225-4483-a0f2-aff776e94a5b-kube-api-access-ztcxr\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.508573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.508533    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:00.521458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.521432    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:00.529660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.529622    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:00.818776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.818736    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"]
Oct 30 13:15:00.821104 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:00.821077    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd508fb_6df4_480e_8a39_1acbeb90d183.slice/crio-3df8051e7d6c50b9599a9515546a035aac8619ab878933ce90d9487fd1aac62b WatchSource:0}: Error finding container 3df8051e7d6c50b9599a9515546a035aac8619ab878933ce90d9487fd1aac62b: Status 404 returned error can't find the container with id 3df8051e7d6c50b9599a9515546a035aac8619ab878933ce90d9487fd1aac62b
Oct 30 13:15:00.845246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.845221    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"]
Oct 30 13:15:00.856138 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:00.856109    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc929656c_5225_4483_a0f2_aff776e94a5b.slice/crio-0d6ff8b70891996b5dd4db9c3fb58a3cf39ca4d0dfb399fb75d705878b21fcf9 WatchSource:0}: Error finding container 0d6ff8b70891996b5dd4db9c3fb58a3cf39ca4d0dfb399fb75d705878b21fcf9: Status 404 returned error can't find the container with id 0d6ff8b70891996b5dd4db9c3fb58a3cf39ca4d0dfb399fb75d705878b21fcf9
Oct 30 13:15:00.858042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:00.858021    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"]
Oct 30 13:15:00.859714 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:00.859682    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf924632e_8a3c_48d2_aa7e_74096322bc7e.slice/crio-0ea72fc32512ffe13e26de4f52b95fcedde3a89150489146ae86bdc52b9ee813 WatchSource:0}: Error finding container 0ea72fc32512ffe13e26de4f52b95fcedde3a89150489146ae86bdc52b9ee813: Status 404 returned error can't find the container with id 0ea72fc32512ffe13e26de4f52b95fcedde3a89150489146ae86bdc52b9ee813
Oct 30 13:15:01.265582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.265550    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" event={"ID":"4cd508fb-6df4-480e-8a39-1acbeb90d183","Type":"ContainerStarted","Data":"3df8051e7d6c50b9599a9515546a035aac8619ab878933ce90d9487fd1aac62b"}
Oct 30 13:15:01.267139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.267107    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" event={"ID":"c929656c-5225-4483-a0f2-aff776e94a5b","Type":"ContainerStarted","Data":"3cb6c42dd374b5b93021b7fc181df28eac61886d55a4def0887ab01207896b4b"}
Oct 30 13:15:01.267270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.267163    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" event={"ID":"c929656c-5225-4483-a0f2-aff776e94a5b","Type":"ContainerStarted","Data":"0d6ff8b70891996b5dd4db9c3fb58a3cf39ca4d0dfb399fb75d705878b21fcf9"}
Oct 30 13:15:01.268660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.268619    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" event={"ID":"f924632e-8a3c-48d2-aa7e-74096322bc7e","Type":"ContainerStarted","Data":"0ea72fc32512ffe13e26de4f52b95fcedde3a89150489146ae86bdc52b9ee813"}
Oct 30 13:15:01.305859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.305802    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" podStartSLOduration=1.3057893059999999 podStartE2EDuration="1.305789306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:01.303976185 +0000 UTC m=+711.564214416" watchObservedRunningTime="2026-10-30 13:15:01.305789306 +0000 UTC m=+711.566027537"
Oct 30 13:15:01.343360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.343307    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" podStartSLOduration=1.3432930459999999 podStartE2EDuration="1.343293046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:01.34116242 +0000 UTC m=+711.601400651" watchObservedRunningTime="2026-10-30 13:15:01.343293046 +0000 UTC m=+711.603531327"
Oct 30 13:15:01.412987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.412957    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:15:01.555712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.555672    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/972f6718-00ca-47e1-9ccb-c52d884ed69f-kube-api-access-87mjq\" (UniqueName: \"kubernetes.io/projected/972f6718-00ca-47e1-9ccb-c52d884ed69f-kube-api-access-87mjq\") pod \"972f6718-00ca-47e1-9ccb-c52d884ed69f\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") "
Oct 30 13:15:01.555908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.555735    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-xks-ca\") pod \"972f6718-00ca-47e1-9ccb-c52d884ed69f\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") "
Oct 30 13:15:01.555908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.555790    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-openshift-ca\") pod \"972f6718-00ca-47e1-9ccb-c52d884ed69f\" (UID: \"972f6718-00ca-47e1-9ccb-c52d884ed69f\") "
Oct 30 13:15:01.556223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.556192    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-openshift-ca" pod "972f6718-00ca-47e1-9ccb-c52d884ed69f" (UID: "972f6718-00ca-47e1-9ccb-c52d884ed69f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:01.557750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.557715    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972f6718-00ca-47e1-9ccb-c52d884ed69f-kube-api-access-87mjq" pod "972f6718-00ca-47e1-9ccb-c52d884ed69f" (UID: "972f6718-00ca-47e1-9ccb-c52d884ed69f"). InnerVolumeSpecName "kube-api-access-87mjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:01.557852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.557791    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-xks-ca" pod "972f6718-00ca-47e1-9ccb-c52d884ed69f" (UID: "972f6718-00ca-47e1-9ccb-c52d884ed69f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:01.657051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.656973    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-87mjq\" (UniqueName: \"kubernetes.io/projected/972f6718-00ca-47e1-9ccb-c52d884ed69f-kube-api-access-87mjq\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.657051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.657003    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:01.657051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:01.657013    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/972f6718-00ca-47e1-9ccb-c52d884ed69f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:02.274727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.274697    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"
Oct 30 13:15:02.274727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.274704    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s" event={"ID":"972f6718-00ca-47e1-9ccb-c52d884ed69f","Type":"ContainerDied","Data":"20064fab619ae8b631cfd4e3863ea5968faf0c09648d3702c96d911bc9373627"}
Oct 30 13:15:02.275181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.274754    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20064fab619ae8b631cfd4e3863ea5968faf0c09648d3702c96d911bc9373627"
Oct 30 13:15:02.276342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.276316    4291 generic.go:356] "Generic (PLEG): container finished" podID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerID="dc5080f2fc0ab76bd5423a8dfa113236b433bb22853da28d332e0b47e1aef94f" exitCode=60
Oct 30 13:15:02.276456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.276399    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" event={"ID":"4cd508fb-6df4-480e-8a39-1acbeb90d183","Type":"ContainerDied","Data":"dc5080f2fc0ab76bd5423a8dfa113236b433bb22853da28d332e0b47e1aef94f"}
Oct 30 13:15:02.276696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.276677    4291 scope.go:122] "RemoveContainer" containerID="dc5080f2fc0ab76bd5423a8dfa113236b433bb22853da28d332e0b47e1aef94f"
Oct 30 13:15:02.278373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.278289    4291 generic.go:356] "Generic (PLEG): container finished" podID="c929656c-5225-4483-a0f2-aff776e94a5b" containerID="3cb6c42dd374b5b93021b7fc181df28eac61886d55a4def0887ab01207896b4b" exitCode=60
Oct 30 13:15:02.278482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.278451    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" event={"ID":"c929656c-5225-4483-a0f2-aff776e94a5b","Type":"ContainerDied","Data":"3cb6c42dd374b5b93021b7fc181df28eac61886d55a4def0887ab01207896b4b"}
Oct 30 13:15:02.278595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.278586    4291 scope.go:122] "RemoveContainer" containerID="3cb6c42dd374b5b93021b7fc181df28eac61886d55a4def0887ab01207896b4b"
Oct 30 13:15:02.280084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.280062    4291 generic.go:356] "Generic (PLEG): container finished" podID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerID="e2e78a25925b7b1c838d9f9753a4d1c7644847bc071d223b29682c768357935e" exitCode=60
Oct 30 13:15:02.280194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.280127    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" event={"ID":"f924632e-8a3c-48d2-aa7e-74096322bc7e","Type":"ContainerDied","Data":"e2e78a25925b7b1c838d9f9753a4d1c7644847bc071d223b29682c768357935e"}
Oct 30 13:15:02.280388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:02.280373    4291 scope.go:122] "RemoveContainer" containerID="e2e78a25925b7b1c838d9f9753a4d1c7644847bc071d223b29682c768357935e"
Oct 30 13:15:03.287252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.287217    4291 generic.go:356] "Generic (PLEG): container finished" podID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerID="bdc7d0ac5740a9ac1ae9c2f10fe910889cb3c3fb2c03a28686f847d6310dfbce" exitCode=60
Oct 30 13:15:03.287721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.287300    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" event={"ID":"4cd508fb-6df4-480e-8a39-1acbeb90d183","Type":"ContainerDied","Data":"bdc7d0ac5740a9ac1ae9c2f10fe910889cb3c3fb2c03a28686f847d6310dfbce"}
Oct 30 13:15:03.287721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.287367    4291 scope.go:122] "RemoveContainer" containerID="dc5080f2fc0ab76bd5423a8dfa113236b433bb22853da28d332e0b47e1aef94f"
Oct 30 13:15:03.287721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.287674    4291 scope.go:122] "RemoveContainer" containerID="bdc7d0ac5740a9ac1ae9c2f10fe910889cb3c3fb2c03a28686f847d6310dfbce"
Oct 30 13:15:03.287996 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:03.287969    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42_odh-ai-gateway-infra(4cd508fb-6df4-480e-8a39-1acbeb90d183)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183"
Oct 30 13:15:03.289226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.289207    4291 generic.go:356] "Generic (PLEG): container finished" podID="c929656c-5225-4483-a0f2-aff776e94a5b" containerID="774a2d1c47df6508c645bec2e25604ebe72e72890be73c1afc0cff22b3628dcc" exitCode=60
Oct 30 13:15:03.289312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.289289    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" event={"ID":"c929656c-5225-4483-a0f2-aff776e94a5b","Type":"ContainerDied","Data":"774a2d1c47df6508c645bec2e25604ebe72e72890be73c1afc0cff22b3628dcc"}
Oct 30 13:15:03.289521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.289508    4291 scope.go:122] "RemoveContainer" containerID="774a2d1c47df6508c645bec2e25604ebe72e72890be73c1afc0cff22b3628dcc"
Oct 30 13:15:03.289778 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:03.289747    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz_odh-ai-gateway-infra(c929656c-5225-4483-a0f2-aff776e94a5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" podUID="c929656c-5225-4483-a0f2-aff776e94a5b"
Oct 30 13:15:03.291173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.291152    4291 generic.go:356] "Generic (PLEG): container finished" podID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerID="7a9d9304b638d29dea41d60f4f17232ca6dab90e0e193817eaaef34c14f3435c" exitCode=60
Oct 30 13:15:03.291284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.291213    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" event={"ID":"f924632e-8a3c-48d2-aa7e-74096322bc7e","Type":"ContainerDied","Data":"7a9d9304b638d29dea41d60f4f17232ca6dab90e0e193817eaaef34c14f3435c"}
Oct 30 13:15:03.291435 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.291419    4291 scope.go:122] "RemoveContainer" containerID="7a9d9304b638d29dea41d60f4f17232ca6dab90e0e193817eaaef34c14f3435c"
Oct 30 13:15:03.291606 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:03.291592    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4_odh-ai-gateway-infra(f924632e-8a3c-48d2-aa7e-74096322bc7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e"
Oct 30 13:15:03.366621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.366591    4291 scope.go:122] "RemoveContainer" containerID="3cb6c42dd374b5b93021b7fc181df28eac61886d55a4def0887ab01207896b4b"
Oct 30 13:15:03.379512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:03.379487    4291 scope.go:122] "RemoveContainer" containerID="e2e78a25925b7b1c838d9f9753a4d1c7644847bc071d223b29682c768357935e"
Oct 30 13:15:04.299322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:04.299294    4291 scope.go:122] "RemoveContainer" containerID="7a9d9304b638d29dea41d60f4f17232ca6dab90e0e193817eaaef34c14f3435c"
Oct 30 13:15:04.299789 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:04.299571    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4_odh-ai-gateway-infra(f924632e-8a3c-48d2-aa7e-74096322bc7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e"
Oct 30 13:15:04.300826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:04.300807    4291 scope.go:122] "RemoveContainer" containerID="bdc7d0ac5740a9ac1ae9c2f10fe910889cb3c3fb2c03a28686f847d6310dfbce"
Oct 30 13:15:04.301038 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:04.301017    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42_odh-ai-gateway-infra(4cd508fb-6df4-480e-8a39-1acbeb90d183)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183"
Oct 30 13:15:04.302205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:04.302188    4291 scope.go:122] "RemoveContainer" containerID="774a2d1c47df6508c645bec2e25604ebe72e72890be73c1afc0cff22b3628dcc"
Oct 30 13:15:04.302390 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:04.302371    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz_odh-ai-gateway-infra(c929656c-5225-4483-a0f2-aff776e94a5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" podUID="c929656c-5225-4483-a0f2-aff776e94a5b"
Oct 30 13:15:05.242079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.242037    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"]
Oct 30 13:15:05.242474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.242461    4291 cpu_manager.go:414] "containerMap: removing container" podUID="972f6718-00ca-47e1-9ccb-c52d884ed69f" containerName="revoke-keys"
Oct 30 13:15:05.242513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.242475    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="972f6718-00ca-47e1-9ccb-c52d884ed69f" containerName="revoke-keys"
Oct 30 13:15:05.242552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.242544    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="972f6718-00ca-47e1-9ccb-c52d884ed69f" containerName="revoke-keys"
Oct 30 13:15:05.247054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.247031    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.263716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.263683    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"]
Oct 30 13:15:05.273180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.273150    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:05.398504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.398455    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.399016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.398657    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.399016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.398777    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpbpb\" (UniqueName: \"kubernetes.io/projected/570c8147-20e0-46df-905b-b5d43038cd74-kube-api-access-mpbpb\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.499770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.499651    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpbpb\" (UniqueName: \"kubernetes.io/projected/570c8147-20e0-46df-905b-b5d43038cd74-kube-api-access-mpbpb\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.499770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.499746    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.500027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.499863    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.500604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.500582    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.502378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.502354    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.520795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.520763    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpbpb\" (UniqueName: \"kubernetes.io/projected/570c8147-20e0-46df-905b-b5d43038cd74-kube-api-access-mpbpb\") pod \"maas-api-revoke-sub-c90ecd6f-86xst\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.560807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.560768    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:05.868810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:05.868779    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"]
Oct 30 13:15:05.870888 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:05.870859    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod570c8147_20e0_46df_905b_b5d43038cd74.slice/crio-f9c3dc0611e11427c8cb745b1082f6d72789779b899063bb692799439d0c66e3 WatchSource:0}: Error finding container f9c3dc0611e11427c8cb745b1082f6d72789779b899063bb692799439d0c66e3: Status 404 returned error can't find the container with id f9c3dc0611e11427c8cb745b1082f6d72789779b899063bb692799439d0c66e3
Oct 30 13:15:06.316786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:06.316750    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst" event={"ID":"570c8147-20e0-46df-905b-b5d43038cd74","Type":"ContainerStarted","Data":"922f26f493504a2584df7800564a36faebc280c2416ad289da2d50a030a435cc"}
Oct 30 13:15:06.316947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:06.316802    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst" event={"ID":"570c8147-20e0-46df-905b-b5d43038cd74","Type":"ContainerStarted","Data":"f9c3dc0611e11427c8cb745b1082f6d72789779b899063bb692799439d0c66e3"}
Oct 30 13:15:06.384976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:06.384921    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst" podStartSLOduration=1.384906559 podStartE2EDuration="1.384906559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:06.382798501 +0000 UTC m=+716.643036737" watchObservedRunningTime="2026-10-30 13:15:06.384906559 +0000 UTC m=+716.645144793"
Oct 30 13:15:07.324305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:07.324267    4291 generic.go:356] "Generic (PLEG): container finished" podID="570c8147-20e0-46df-905b-b5d43038cd74" containerID="922f26f493504a2584df7800564a36faebc280c2416ad289da2d50a030a435cc" exitCode=0
Oct 30 13:15:07.324764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:07.324362    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst" event={"ID":"570c8147-20e0-46df-905b-b5d43038cd74","Type":"ContainerDied","Data":"922f26f493504a2584df7800564a36faebc280c2416ad289da2d50a030a435cc"}
Oct 30 13:15:08.210462 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:08.210426    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:15:08.469033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.468963    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:08.529474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.529432    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-xks-ca\") pod \"570c8147-20e0-46df-905b-b5d43038cd74\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") "
Oct 30 13:15:08.529673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.529492    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-openshift-ca\") pod \"570c8147-20e0-46df-905b-b5d43038cd74\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") "
Oct 30 13:15:08.529673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.529567    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/570c8147-20e0-46df-905b-b5d43038cd74-kube-api-access-mpbpb\" (UniqueName: \"kubernetes.io/projected/570c8147-20e0-46df-905b-b5d43038cd74-kube-api-access-mpbpb\") pod \"570c8147-20e0-46df-905b-b5d43038cd74\" (UID: \"570c8147-20e0-46df-905b-b5d43038cd74\") "
Oct 30 13:15:08.529979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.529945    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-openshift-ca" pod "570c8147-20e0-46df-905b-b5d43038cd74" (UID: "570c8147-20e0-46df-905b-b5d43038cd74"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:08.531775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.531748    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/570c8147-20e0-46df-905b-b5d43038cd74-kube-api-access-mpbpb" pod "570c8147-20e0-46df-905b-b5d43038cd74" (UID: "570c8147-20e0-46df-905b-b5d43038cd74"). InnerVolumeSpecName "kube-api-access-mpbpb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:08.531910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.531818    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-xks-ca" pod "570c8147-20e0-46df-905b-b5d43038cd74" (UID: "570c8147-20e0-46df-905b-b5d43038cd74"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:08.631324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.631283    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpbpb\" (UniqueName: \"kubernetes.io/projected/570c8147-20e0-46df-905b-b5d43038cd74-kube-api-access-mpbpb\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:08.631324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.631317    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:08.631324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:08.631328    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/570c8147-20e0-46df-905b-b5d43038cd74-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:09.336790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:09.336760    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"
Oct 30 13:15:09.336790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:09.336770    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst" event={"ID":"570c8147-20e0-46df-905b-b5d43038cd74","Type":"ContainerDied","Data":"f9c3dc0611e11427c8cb745b1082f6d72789779b899063bb692799439d0c66e3"}
Oct 30 13:15:09.337008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:09.336842    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9c3dc0611e11427c8cb745b1082f6d72789779b899063bb692799439d0c66e3"
Oct 30 13:15:13.871517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:13.871482    4291 scope.go:122] "RemoveContainer" containerID="774a2d1c47df6508c645bec2e25604ebe72e72890be73c1afc0cff22b3628dcc"
Oct 30 13:15:14.398560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:14.398523    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" event={"ID":"c929656c-5225-4483-a0f2-aff776e94a5b","Type":"ContainerStarted","Data":"35872a59c52defeb51ed9c64811d5589eed98e3a252cdad58d973ec1fb8a313a"}
Oct 30 13:15:15.010537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.010505    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:15:15.406436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.406402    4291 generic.go:356] "Generic (PLEG): container finished" podID="c929656c-5225-4483-a0f2-aff776e94a5b" containerID="35872a59c52defeb51ed9c64811d5589eed98e3a252cdad58d973ec1fb8a313a" exitCode=60
Oct 30 13:15:15.406610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.406448    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" event={"ID":"c929656c-5225-4483-a0f2-aff776e94a5b","Type":"ContainerDied","Data":"35872a59c52defeb51ed9c64811d5589eed98e3a252cdad58d973ec1fb8a313a"}
Oct 30 13:15:15.406610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.406502    4291 scope.go:122] "RemoveContainer" containerID="774a2d1c47df6508c645bec2e25604ebe72e72890be73c1afc0cff22b3628dcc"
Oct 30 13:15:15.406944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.406925    4291 scope.go:122] "RemoveContainer" containerID="35872a59c52defeb51ed9c64811d5589eed98e3a252cdad58d973ec1fb8a313a"
Oct 30 13:15:15.407198 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:15.407175    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz_odh-ai-gateway-infra(c929656c-5225-4483-a0f2-aff776e94a5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" podUID="c929656c-5225-4483-a0f2-aff776e94a5b"
Oct 30 13:15:15.452791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.452759    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"]
Oct 30 13:15:15.680969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.680897    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"]
Oct 30 13:15:15.681348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.681335    4291 cpu_manager.go:414] "containerMap: removing container" podUID="570c8147-20e0-46df-905b-b5d43038cd74" containerName="revoke-keys"
Oct 30 13:15:15.681389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.681351    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="570c8147-20e0-46df-905b-b5d43038cd74" containerName="revoke-keys"
Oct 30 13:15:15.681431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.681421    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="570c8147-20e0-46df-905b-b5d43038cd74" containerName="revoke-keys"
Oct 30 13:15:15.686032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.686017    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.700521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.700496    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"]
Oct 30 13:15:15.701486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.701470    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:15.715008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.714985    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.715105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.715033    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mndnb\" (UniqueName: \"kubernetes.io/projected/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-kube-api-access-mndnb\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.715105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.715058    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.815894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.815861    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mndnb\" (UniqueName: \"kubernetes.io/projected/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-kube-api-access-mndnb\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.816096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.815914    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.816096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.816067    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.816587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.816555    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.818368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.818350    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.843790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.843763    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mndnb\" (UniqueName: \"kubernetes.io/projected/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-kube-api-access-mndnb\") pod \"maas-api-revoke-sub-3bb9da49-xhlzd\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:15.998758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:15.998676    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:16.353961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.353917    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"]
Oct 30 13:15:16.356024 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:16.355978    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ee5308a_adbc_49d3_89f4_46cc2cc4f400.slice/crio-d80f32462ce65cffd0257775b5a59ffddccad0dcc006fb3173ea81bb9dc61939 WatchSource:0}: Error finding container d80f32462ce65cffd0257775b5a59ffddccad0dcc006fb3173ea81bb9dc61939: Status 404 returned error can't find the container with id d80f32462ce65cffd0257775b5a59ffddccad0dcc006fb3173ea81bb9dc61939
Oct 30 13:15:16.416233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.416149    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd" event={"ID":"9ee5308a-adbc-49d3-89f4-46cc2cc4f400","Type":"ContainerStarted","Data":"d80f32462ce65cffd0257775b5a59ffddccad0dcc006fb3173ea81bb9dc61939"}
Oct 30 13:15:16.551490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.551462    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:16.625987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.625890    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c929656c-5225-4483-a0f2-aff776e94a5b-kube-api-access-ztcxr\" (UniqueName: \"kubernetes.io/projected/c929656c-5225-4483-a0f2-aff776e94a5b-kube-api-access-ztcxr\") pod \"c929656c-5225-4483-a0f2-aff776e94a5b\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") "
Oct 30 13:15:16.626142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.626033    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c929656c-5225-4483-a0f2-aff776e94a5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c929656c-5225-4483-a0f2-aff776e94a5b-maas-api-service-ca\") pod \"c929656c-5225-4483-a0f2-aff776e94a5b\" (UID: \"c929656c-5225-4483-a0f2-aff776e94a5b\") "
Oct 30 13:15:16.626612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.626586    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c929656c-5225-4483-a0f2-aff776e94a5b-maas-api-service-ca" pod "c929656c-5225-4483-a0f2-aff776e94a5b" (UID: "c929656c-5225-4483-a0f2-aff776e94a5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:16.628717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.628689    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c929656c-5225-4483-a0f2-aff776e94a5b-kube-api-access-ztcxr" pod "c929656c-5225-4483-a0f2-aff776e94a5b" (UID: "c929656c-5225-4483-a0f2-aff776e94a5b"). InnerVolumeSpecName "kube-api-access-ztcxr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:16.727079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.727045    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c929656c-5225-4483-a0f2-aff776e94a5b-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:16.727079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:16.727074    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztcxr\" (UniqueName: \"kubernetes.io/projected/c929656c-5225-4483-a0f2-aff776e94a5b-kube-api-access-ztcxr\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:17.434123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.431493    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"
Oct 30 13:15:17.434123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.432033    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz" event={"ID":"c929656c-5225-4483-a0f2-aff776e94a5b","Type":"ContainerDied","Data":"0d6ff8b70891996b5dd4db9c3fb58a3cf39ca4d0dfb399fb75d705878b21fcf9"}
Oct 30 13:15:17.434123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.432099    4291 scope.go:122] "RemoveContainer" containerID="35872a59c52defeb51ed9c64811d5589eed98e3a252cdad58d973ec1fb8a313a"
Oct 30 13:15:17.438468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.438394    4291 generic.go:356] "Generic (PLEG): container finished" podID="9ee5308a-adbc-49d3-89f4-46cc2cc4f400" containerID="353a9089143fcad5e1a3655b9242be8387dfe3dd00154174a33329d3a5f33e91" exitCode=0
Oct 30 13:15:17.438749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.438438    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd" event={"ID":"9ee5308a-adbc-49d3-89f4-46cc2cc4f400","Type":"ContainerDied","Data":"353a9089143fcad5e1a3655b9242be8387dfe3dd00154174a33329d3a5f33e91"}
Oct 30 13:15:17.536041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.535970    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"]
Oct 30 13:15:17.539117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.539087    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889435-s7rjz"]
Oct 30 13:15:17.871395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.871357    4291 scope.go:122] "RemoveContainer" containerID="7a9d9304b638d29dea41d60f4f17232ca6dab90e0e193817eaaef34c14f3435c"
Oct 30 13:15:17.872474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.872451    4291 scope.go:122] "RemoveContainer" containerID="bdc7d0ac5740a9ac1ae9c2f10fe910889cb3c3fb2c03a28686f847d6310dfbce"
Oct 30 13:15:17.930505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:17.930371    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" path="/var/lib/kubelet/pods/c929656c-5225-4483-a0f2-aff776e94a5b/volumes"
Oct 30 13:15:18.446388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.446355    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" event={"ID":"f924632e-8a3c-48d2-aa7e-74096322bc7e","Type":"ContainerStarted","Data":"b8633970397c16c2db2d773af894e9dbe088f368ce6920758058ba7a9d59f4bb"}
Oct 30 13:15:18.448321 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.448259    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" event={"ID":"4cd508fb-6df4-480e-8a39-1acbeb90d183","Type":"ContainerStarted","Data":"ea9b73552abf405d693222fff744d71ede2dce487dc2bf9732a696ee79b47c4e"}
Oct 30 13:15:18.577123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.576788    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" podStartSLOduration=18.576765284 podStartE2EDuration="18.576765284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:18.564021722 +0000 UTC m=+728.824259955" watchObservedRunningTime="2026-10-30 13:15:18.576765284 +0000 UTC m=+728.837003518"
Oct 30 13:15:18.782930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.782901    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:18.858938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.858908    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-xks-ca\") pod \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") "
Oct 30 13:15:18.859102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.859066    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-openshift-ca\") pod \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") "
Oct 30 13:15:18.859193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.859123    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-kube-api-access-mndnb\" (UniqueName: \"kubernetes.io/projected/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-kube-api-access-mndnb\") pod \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\" (UID: \"9ee5308a-adbc-49d3-89f4-46cc2cc4f400\") "
Oct 30 13:15:18.859768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.859717    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-openshift-ca" pod "9ee5308a-adbc-49d3-89f4-46cc2cc4f400" (UID: "9ee5308a-adbc-49d3-89f4-46cc2cc4f400"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:18.860268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.860242    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.861414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.861392    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-xks-ca" pod "9ee5308a-adbc-49d3-89f4-46cc2cc4f400" (UID: "9ee5308a-adbc-49d3-89f4-46cc2cc4f400"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:15:18.861950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.861923    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-kube-api-access-mndnb" pod "9ee5308a-adbc-49d3-89f4-46cc2cc4f400" (UID: "9ee5308a-adbc-49d3-89f4-46cc2cc4f400"). InnerVolumeSpecName "kube-api-access-mndnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:18.962096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.962066    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:18.962096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:18.962098    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mndnb\" (UniqueName: \"kubernetes.io/projected/9ee5308a-adbc-49d3-89f4-46cc2cc4f400-kube-api-access-mndnb\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:19.455652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.455607    4291 generic.go:356] "Generic (PLEG): container finished" podID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerID="ea9b73552abf405d693222fff744d71ede2dce487dc2bf9732a696ee79b47c4e" exitCode=60
Oct 30 13:15:19.456107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.455687    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" event={"ID":"4cd508fb-6df4-480e-8a39-1acbeb90d183","Type":"ContainerDied","Data":"ea9b73552abf405d693222fff744d71ede2dce487dc2bf9732a696ee79b47c4e"}
Oct 30 13:15:19.456107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.455752    4291 scope.go:122] "RemoveContainer" containerID="bdc7d0ac5740a9ac1ae9c2f10fe910889cb3c3fb2c03a28686f847d6310dfbce"
Oct 30 13:15:19.456107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.456063    4291 scope.go:122] "RemoveContainer" containerID="ea9b73552abf405d693222fff744d71ede2dce487dc2bf9732a696ee79b47c4e"
Oct 30 13:15:19.456356 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:19.456337    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42_odh-ai-gateway-infra(4cd508fb-6df4-480e-8a39-1acbeb90d183)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183"
Oct 30 13:15:19.457052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.457039    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"
Oct 30 13:15:19.457119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.457072    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd" event={"ID":"9ee5308a-adbc-49d3-89f4-46cc2cc4f400","Type":"ContainerDied","Data":"d80f32462ce65cffd0257775b5a59ffddccad0dcc006fb3173ea81bb9dc61939"}
Oct 30 13:15:19.457171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.457124    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d80f32462ce65cffd0257775b5a59ffddccad0dcc006fb3173ea81bb9dc61939"
Oct 30 13:15:19.458906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.458888    4291 generic.go:356] "Generic (PLEG): container finished" podID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerID="b8633970397c16c2db2d773af894e9dbe088f368ce6920758058ba7a9d59f4bb" exitCode=60
Oct 30 13:15:19.458996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.458961    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" event={"ID":"f924632e-8a3c-48d2-aa7e-74096322bc7e","Type":"ContainerDied","Data":"b8633970397c16c2db2d773af894e9dbe088f368ce6920758058ba7a9d59f4bb"}
Oct 30 13:15:19.459268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.459253    4291 scope.go:122] "RemoveContainer" containerID="b8633970397c16c2db2d773af894e9dbe088f368ce6920758058ba7a9d59f4bb"
Oct 30 13:15:19.459466 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:19.459449    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4_odh-ai-gateway-infra(f924632e-8a3c-48d2-aa7e-74096322bc7e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e"
Oct 30 13:15:19.467862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.467845    4291 scope.go:122] "RemoveContainer" containerID="7a9d9304b638d29dea41d60f4f17232ca6dab90e0e193817eaaef34c14f3435c"
Oct 30 13:15:19.499797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.499771    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"]
Oct 30 13:15:19.571746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:19.571716    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"]
Oct 30 13:15:20.615330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.615290    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:20.642715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.642693    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:20.679300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.679268    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cd508fb-6df4-480e-8a39-1acbeb90d183-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd508fb-6df4-480e-8a39-1acbeb90d183-maas-api-service-ca\") pod \"4cd508fb-6df4-480e-8a39-1acbeb90d183\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") "
Oct 30 13:15:20.679485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.679398    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f924632e-8a3c-48d2-aa7e-74096322bc7e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f924632e-8a3c-48d2-aa7e-74096322bc7e-maas-api-service-ca\") pod \"f924632e-8a3c-48d2-aa7e-74096322bc7e\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") "
Oct 30 13:15:20.679485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.679431    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f924632e-8a3c-48d2-aa7e-74096322bc7e-kube-api-access-d4hl5\" (UniqueName: \"kubernetes.io/projected/f924632e-8a3c-48d2-aa7e-74096322bc7e-kube-api-access-d4hl5\") pod \"f924632e-8a3c-48d2-aa7e-74096322bc7e\" (UID: \"f924632e-8a3c-48d2-aa7e-74096322bc7e\") "
Oct 30 13:15:20.679589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.679569    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cd508fb-6df4-480e-8a39-1acbeb90d183-kube-api-access-qxqvg\" (UniqueName: \"kubernetes.io/projected/4cd508fb-6df4-480e-8a39-1acbeb90d183-kube-api-access-qxqvg\") pod \"4cd508fb-6df4-480e-8a39-1acbeb90d183\" (UID: \"4cd508fb-6df4-480e-8a39-1acbeb90d183\") "
Oct 30 13:15:20.679791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.679762    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f924632e-8a3c-48d2-aa7e-74096322bc7e-maas-api-service-ca" pod "f924632e-8a3c-48d2-aa7e-74096322bc7e" (UID: "f924632e-8a3c-48d2-aa7e-74096322bc7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.679936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.679919    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f924632e-8a3c-48d2-aa7e-74096322bc7e-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.680964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.680938    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cd508fb-6df4-480e-8a39-1acbeb90d183-maas-api-service-ca" pod "4cd508fb-6df4-480e-8a39-1acbeb90d183" (UID: "4cd508fb-6df4-480e-8a39-1acbeb90d183"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:15:20.681923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.681893    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cd508fb-6df4-480e-8a39-1acbeb90d183-kube-api-access-qxqvg" pod "4cd508fb-6df4-480e-8a39-1acbeb90d183" (UID: "4cd508fb-6df4-480e-8a39-1acbeb90d183"). InnerVolumeSpecName "kube-api-access-qxqvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.682407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.682386    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f924632e-8a3c-48d2-aa7e-74096322bc7e-kube-api-access-d4hl5" pod "f924632e-8a3c-48d2-aa7e-74096322bc7e" (UID: "f924632e-8a3c-48d2-aa7e-74096322bc7e"). InnerVolumeSpecName "kube-api-access-d4hl5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:15:20.781060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.780984    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxqvg\" (UniqueName: \"kubernetes.io/projected/4cd508fb-6df4-480e-8a39-1acbeb90d183-kube-api-access-qxqvg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.781060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.781008    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd508fb-6df4-480e-8a39-1acbeb90d183-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:20.781060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:20.781018    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4hl5\" (UniqueName: \"kubernetes.io/projected/f924632e-8a3c-48d2-aa7e-74096322bc7e-kube-api-access-d4hl5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:15:21.474368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.474340    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"
Oct 30 13:15:21.474572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.474378    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42" event={"ID":"4cd508fb-6df4-480e-8a39-1acbeb90d183","Type":"ContainerDied","Data":"3df8051e7d6c50b9599a9515546a035aac8619ab878933ce90d9487fd1aac62b"}
Oct 30 13:15:21.474572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.474441    4291 scope.go:122] "RemoveContainer" containerID="ea9b73552abf405d693222fff744d71ede2dce487dc2bf9732a696ee79b47c4e"
Oct 30 13:15:21.475654 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.475580    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4" event={"ID":"f924632e-8a3c-48d2-aa7e-74096322bc7e","Type":"ContainerDied","Data":"0ea72fc32512ffe13e26de4f52b95fcedde3a89150489146ae86bdc52b9ee813"}
Oct 30 13:15:21.475732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.475670    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"
Oct 30 13:15:21.484111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.484091    4291 scope.go:122] "RemoveContainer" containerID="b8633970397c16c2db2d773af894e9dbe088f368ce6920758058ba7a9d59f4bb"
Oct 30 13:15:21.528971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.528950    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"]
Oct 30 13:15:21.540247 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.540225    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889435-9gc42"]
Oct 30 13:15:21.571655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.571616    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"]
Oct 30 13:15:21.574894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.574872    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-9bf8ad-29889435-ccwd4"]
Oct 30 13:15:21.876464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.876429    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" path="/var/lib/kubelet/pods/4cd508fb-6df4-480e-8a39-1acbeb90d183/volumes"
Oct 30 13:15:21.876877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:21.876862    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" path="/var/lib/kubelet/pods/f924632e-8a3c-48d2-aa7e-74096322bc7e/volumes"
Oct 30 13:15:30.620208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620175    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"]
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620731    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620751    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620764    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620771    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620783    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9ee5308a-adbc-49d3-89f4-46cc2cc4f400" containerName="revoke-keys"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620790    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9ee5308a-adbc-49d3-89f4-46cc2cc4f400" containerName="revoke-keys"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620806    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620813    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620825    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620832    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620841    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620847    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620878    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620884    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620900    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.620906    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621012    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621022    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621031    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621040    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621048    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621056    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9ee5308a-adbc-49d3-89f4-46cc2cc4f400" containerName="revoke-keys"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621067    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621073    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cd508fb-6df4-480e-8a39-1acbeb90d183" containerName="cleanup"
Oct 30 13:15:30.621239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.621083    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:30.624763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.624739    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.640066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.640039    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"]
Oct 30 13:15:30.649687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.649663    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"distinct-w3-5852ff-kserve-self-signed-certs\""
Oct 30 13:15:30.701402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.701368    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-tmp-dir\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.701593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.701430    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tkh4\" (UniqueName: \"kubernetes.io/projected/46c01365-1530-4b25-829b-072cdbadb706-kube-api-access-2tkh4\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.701696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.701588    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-home\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.701696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.701627    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-dshm\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.701785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.701761    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c01365-1530-4b25-829b-072cdbadb706-tls-certs\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.701836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.701818    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-model-cache\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803029    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-model-cache\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803097    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-tmp-dir\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803140    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tkh4\" (UniqueName: \"kubernetes.io/projected/46c01365-1530-4b25-829b-072cdbadb706-kube-api-access-2tkh4\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803226    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-home\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803250    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-dshm\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803299    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c01365-1530-4b25-829b-072cdbadb706-tls-certs\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803530    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-tmp-dir\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.803770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803599    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-home\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.804064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.803987    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-model-cache\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.808246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.808221    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c01365-1530-4b25-829b-072cdbadb706-tls-certs\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.808534 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.808513    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-dshm\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.827125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.827083    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tkh4\" (UniqueName: \"kubernetes.io/projected/46c01365-1530-4b25-829b-072cdbadb706-kube-api-access-2tkh4\") pod \"distinct-w3-5852ff-kserve-678897f6b4-lc57p\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:30.939761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:30.939662    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:31.266909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:31.266756    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"]
Oct 30 13:15:31.269697 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:31.269671    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46c01365_1530_4b25_829b_072cdbadb706.slice/crio-3c0b70635f2c8a689159ccad05798c1af087560c5fd2975f0eddfff86119baf0 WatchSource:0}: Error finding container 3c0b70635f2c8a689159ccad05798c1af087560c5fd2975f0eddfff86119baf0: Status 404 returned error can't find the container with id 3c0b70635f2c8a689159ccad05798c1af087560c5fd2975f0eddfff86119baf0
Oct 30 13:15:31.550850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:31.550758    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" event={"ID":"46c01365-1530-4b25-829b-072cdbadb706","Type":"ContainerStarted","Data":"39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a"}
Oct 30 13:15:31.550850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:31.550801    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" event={"ID":"46c01365-1530-4b25-829b-072cdbadb706","Type":"ContainerStarted","Data":"3c0b70635f2c8a689159ccad05798c1af087560c5fd2975f0eddfff86119baf0"}
Oct 30 13:15:31.592468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:31.592416    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podStartSLOduration=1.59240228 podStartE2EDuration="1.59240228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:31.590137535 +0000 UTC m=+741.850375770" watchObservedRunningTime="2026-10-30 13:15:31.59240228 +0000 UTC m=+741.852640540"
Oct 30 13:15:31.939736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:31.939702    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:36.319719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:36.319681    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:15:38.248143 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:15:38.248103    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:15:40.940711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:40.940666    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:40.955213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:40.955182    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:41.646827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:41.646797    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:15:45.304614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.304579    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"]
Oct 30 13:15:45.305228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.305206    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:45.305273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.305232    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c929656c-5225-4483-a0f2-aff776e94a5b" containerName="cleanup"
Oct 30 13:15:45.305273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.305257    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:45.305273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.305265    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:45.305412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.305400    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f924632e-8a3c-48d2-aa7e-74096322bc7e" containerName="cleanup"
Oct 30 13:15:45.310222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.310200    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.323586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.323563    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"]
Oct 30 13:15:45.333459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.333439    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"openshift-service-ca.crt\""
Oct 30 13:15:45.333578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.333527    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"default-dockercfg-4lf5d\""
Oct 30 13:15:45.336337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.336320    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"kube-root-ca.crt\""
Oct 30 13:15:45.336403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.336387    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-9bf8ad\"/\"e2e-conflict-a-dfdb6918-kserve-self-signed-certs\""
Oct 30 13:15:45.378495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.378471    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-dshm\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.378668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.378513    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pm6s\" (UniqueName: \"kubernetes.io/projected/96115c1b-7ade-4f59-ad95-afdf3ba11a01-kube-api-access-8pm6s\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.378668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.378545    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tls-certs\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.378668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.378628    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tmp-dir\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.378846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.378745    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-home\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.378846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.378793    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-model-cache\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.479533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.479488    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tls-certs\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.479533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.479541    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tmp-dir\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.479789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.479627    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-home\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.479789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.479700    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-model-cache\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.479789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.479754    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-dshm\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.479911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.479824    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pm6s\" (UniqueName: \"kubernetes.io/projected/96115c1b-7ade-4f59-ad95-afdf3ba11a01-kube-api-access-8pm6s\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.480436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.480408    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-home\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.480726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.480704    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-model-cache\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.481132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.481049    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tmp-dir\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.482966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.482939    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tls-certs\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.487960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.487937    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-dshm\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.499910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.499881    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pm6s\" (UniqueName: \"kubernetes.io/projected/96115c1b-7ade-4f59-ad95-afdf3ba11a01-kube-api-access-8pm6s\") pod \"e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") " pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.623839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.623807    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:45.923881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:45.923848    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"]
Oct 30 13:15:45.925430 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:45.925403    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96115c1b_7ade_4f59_ad95_afdf3ba11a01.slice/crio-ac42197a9592a90ad6a8838ad629107db6cdf8dd548b59e8be0636212126d052 WatchSource:0}: Error finding container ac42197a9592a90ad6a8838ad629107db6cdf8dd548b59e8be0636212126d052: Status 404 returned error can't find the container with id ac42197a9592a90ad6a8838ad629107db6cdf8dd548b59e8be0636212126d052
Oct 30 13:15:46.665688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:46.665630    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x" event={"ID":"96115c1b-7ade-4f59-ad95-afdf3ba11a01","Type":"ContainerStarted","Data":"351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de"}
Oct 30 13:15:46.666077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:46.665696    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x" event={"ID":"96115c1b-7ade-4f59-ad95-afdf3ba11a01","Type":"ContainerStarted","Data":"ac42197a9592a90ad6a8838ad629107db6cdf8dd548b59e8be0636212126d052"}
Oct 30 13:15:46.704804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:46.704758    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x" podStartSLOduration=1.704745011 podStartE2EDuration="1.704745011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:46.703604575 +0000 UTC m=+756.963842808" watchObservedRunningTime="2026-10-30 13:15:46.704745011 +0000 UTC m=+756.964983245"
Oct 30 13:15:47.624074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:47.624037    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:55.624181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.624085    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:55.637607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.637579    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:55.658224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.658195    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"]
Oct 30 13:15:55.662012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.661994    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.676949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.676914    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5852ff\"/\"distinct-2-w3-5852ff-kserve-self-signed-certs\""
Oct 30 13:15:55.677776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.677750    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"]
Oct 30 13:15:55.679689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.679663    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-tmp-dir\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.679861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.679795    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-model-cache\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.679861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.679849    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5779532-3650-4896-95ee-35329a605600-tls-certs\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.679980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.679867    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-dshm\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.679980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.679897    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-home\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.679980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.679956    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tm2f\" (UniqueName: \"kubernetes.io/projected/d5779532-3650-4896-95ee-35329a605600-kube-api-access-5tm2f\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.740216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.740181    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:15:55.781360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.781315    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tm2f\" (UniqueName: \"kubernetes.io/projected/d5779532-3650-4896-95ee-35329a605600-kube-api-access-5tm2f\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.781567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.781391    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-tmp-dir\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.781567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.781495    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-model-cache\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.781567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.781557    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5779532-3650-4896-95ee-35329a605600-tls-certs\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.781799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.781583    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-dshm\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.781799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.781618    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-home\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.781931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.781903    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-tmp-dir\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.782092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.782068    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-model-cache\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.782233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.782004    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-home\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.784554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.784521    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-dshm\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.795977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.795936    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5779532-3650-4896-95ee-35329a605600-tls-certs\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.804774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.804732    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tm2f\" (UniqueName: \"kubernetes.io/projected/d5779532-3650-4896-95ee-35329a605600-kube-api-access-5tm2f\") pod \"distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") " pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:55.981927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:55.981844    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:56.300871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:56.300841    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"]
Oct 30 13:15:56.302517 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:56.302492    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5779532_3650_4896_95ee_35329a605600.slice/crio-fdd4335a5dca9b55a8e909831c9ca213ce1702335746fb6579f1a7697be81358 WatchSource:0}: Error finding container fdd4335a5dca9b55a8e909831c9ca213ce1702335746fb6579f1a7697be81358: Status 404 returned error can't find the container with id fdd4335a5dca9b55a8e909831c9ca213ce1702335746fb6579f1a7697be81358
Oct 30 13:15:56.735733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:56.735697    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" event={"ID":"d5779532-3650-4896-95ee-35329a605600","Type":"ContainerStarted","Data":"e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9"}
Oct 30 13:15:56.736218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:56.735748    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" event={"ID":"d5779532-3650-4896-95ee-35329a605600","Type":"ContainerStarted","Data":"fdd4335a5dca9b55a8e909831c9ca213ce1702335746fb6579f1a7697be81358"}
Oct 30 13:15:56.776897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:56.776842    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podStartSLOduration=1.776827145 podStartE2EDuration="1.776827145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:56.774478857 +0000 UTC m=+767.034717090" watchObservedRunningTime="2026-10-30 13:15:56.776827145 +0000 UTC m=+767.037065378"
Oct 30 13:15:56.982367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:56.982323    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:15:57.400299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.400266    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"]
Oct 30 13:15:57.403602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.403587    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.420446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.420416    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"]
Oct 30 13:15:57.430043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.430015    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:15:57.430043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.430016    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:15:57.430208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.430074    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:15:57.430208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.430153    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:15:57.498756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.498723    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9cgw\" (UniqueName: \"kubernetes.io/projected/7204027e-9bf3-44fd-b9b0-9726409daefb-kube-api-access-v9cgw\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.498930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.498772    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.498930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.498790    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.599757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.599724    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9cgw\" (UniqueName: \"kubernetes.io/projected/7204027e-9bf3-44fd-b9b0-9726409daefb-kube-api-access-v9cgw\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.599942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.599803    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.599994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.599955    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.600443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.600423    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.602329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.602306    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.619004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.618976    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9cgw\" (UniqueName: \"kubernetes.io/projected/7204027e-9bf3-44fd-b9b0-9726409daefb-kube-api-access-v9cgw\") pod \"maas-api-revoke-sub-38b3eed7-fqmrt\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:57.717180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:57.717094    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:15:58.024740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:58.022364    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"]
Oct 30 13:15:58.028486 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:15:58.028461    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7204027e_9bf3_44fd_b9b0_9726409daefb.slice/crio-4f8218b8e39920e39e2cf7a75a09d4de764754bea26c7b94b3eabd2221ead20b WatchSource:0}: Error finding container 4f8218b8e39920e39e2cf7a75a09d4de764754bea26c7b94b3eabd2221ead20b: Status 404 returned error can't find the container with id 4f8218b8e39920e39e2cf7a75a09d4de764754bea26c7b94b3eabd2221ead20b
Oct 30 13:15:58.750197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:58.750104    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt" event={"ID":"7204027e-9bf3-44fd-b9b0-9726409daefb","Type":"ContainerStarted","Data":"dc303d86ccf66a3ac12de2ffb88ff56e092975ef309b11c3c75c6989ef72e024"}
Oct 30 13:15:58.750197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:58.750161    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt" event={"ID":"7204027e-9bf3-44fd-b9b0-9726409daefb","Type":"ContainerStarted","Data":"4f8218b8e39920e39e2cf7a75a09d4de764754bea26c7b94b3eabd2221ead20b"}
Oct 30 13:15:58.788431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:58.788383    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt" podStartSLOduration=1.788369167 podStartE2EDuration="1.788369167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:15:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:15:58.78743357 +0000 UTC m=+769.047671807" watchObservedRunningTime="2026-10-30 13:15:58.788369167 +0000 UTC m=+769.048607401"
Oct 30 13:15:59.758565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:59.758528    4291 generic.go:356] "Generic (PLEG): container finished" podID="7204027e-9bf3-44fd-b9b0-9726409daefb" containerID="dc303d86ccf66a3ac12de2ffb88ff56e092975ef309b11c3c75c6989ef72e024" exitCode=0
Oct 30 13:15:59.759076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:15:59.758605    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt" event={"ID":"7204027e-9bf3-44fd-b9b0-9726409daefb","Type":"ContainerDied","Data":"dc303d86ccf66a3ac12de2ffb88ff56e092975ef309b11c3c75c6989ef72e024"}
Oct 30 13:16:00.903016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.902988    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:16:00.935955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.935922    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7204027e-9bf3-44fd-b9b0-9726409daefb-kube-api-access-v9cgw\" (UniqueName: \"kubernetes.io/projected/7204027e-9bf3-44fd-b9b0-9726409daefb-kube-api-access-v9cgw\") pod \"7204027e-9bf3-44fd-b9b0-9726409daefb\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") "
Oct 30 13:16:00.936122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.936045    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-openshift-ca\") pod \"7204027e-9bf3-44fd-b9b0-9726409daefb\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") "
Oct 30 13:16:00.936201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.936183    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-xks-ca\") pod \"7204027e-9bf3-44fd-b9b0-9726409daefb\" (UID: \"7204027e-9bf3-44fd-b9b0-9726409daefb\") "
Oct 30 13:16:00.936434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.936410    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-openshift-ca" pod "7204027e-9bf3-44fd-b9b0-9726409daefb" (UID: "7204027e-9bf3-44fd-b9b0-9726409daefb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:00.936551 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.936535    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:00.938166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.938139    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-xks-ca" pod "7204027e-9bf3-44fd-b9b0-9726409daefb" (UID: "7204027e-9bf3-44fd-b9b0-9726409daefb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:00.938277 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:00.938254    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7204027e-9bf3-44fd-b9b0-9726409daefb-kube-api-access-v9cgw" pod "7204027e-9bf3-44fd-b9b0-9726409daefb" (UID: "7204027e-9bf3-44fd-b9b0-9726409daefb"). InnerVolumeSpecName "kube-api-access-v9cgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:01.037725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:01.037630    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9cgw\" (UniqueName: \"kubernetes.io/projected/7204027e-9bf3-44fd-b9b0-9726409daefb-kube-api-access-v9cgw\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:01.037725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:01.037677    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7204027e-9bf3-44fd-b9b0-9726409daefb-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:01.771813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:01.771776    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt" event={"ID":"7204027e-9bf3-44fd-b9b0-9726409daefb","Type":"ContainerDied","Data":"4f8218b8e39920e39e2cf7a75a09d4de764754bea26c7b94b3eabd2221ead20b"}
Oct 30 13:16:01.771813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:01.771808    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"
Oct 30 13:16:01.772050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:01.771864    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f8218b8e39920e39e2cf7a75a09d4de764754bea26c7b94b3eabd2221ead20b"
Oct 30 13:16:02.820100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:02.820062    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:16:05.982844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:05.982806    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:16:05.995580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:05.995553    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:16:06.810527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:06.810498    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:16:08.210057 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:16:08.210023    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:16:09.819393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.819361    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"]
Oct 30 13:16:09.819819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.819805    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7204027e-9bf3-44fd-b9b0-9726409daefb" containerName="revoke-keys"
Oct 30 13:16:09.819863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.819820    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7204027e-9bf3-44fd-b9b0-9726409daefb" containerName="revoke-keys"
Oct 30 13:16:09.819903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.819893    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7204027e-9bf3-44fd-b9b0-9726409daefb" containerName="revoke-keys"
Oct 30 13:16:09.823179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.823162    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:09.841569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.841541    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"]
Oct 30 13:16:09.850319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.850285    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:16:09.850319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.850312    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:16:09.850525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.850341    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:16:09.851501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.851485    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:16:09.932847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.932818    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:09.932847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.932850    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:09.933055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:09.932960    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frt6v\" (UniqueName: \"kubernetes.io/projected/8b398c60-0b41-4ecc-80d3-2e336559a338-kube-api-access-frt6v\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.034022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.033984    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frt6v\" (UniqueName: \"kubernetes.io/projected/8b398c60-0b41-4ecc-80d3-2e336559a338-kube-api-access-frt6v\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.034205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.034123    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.034205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.034143    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.034989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.034962    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.036576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.036555    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.054174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.054144    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-frt6v\" (UniqueName: \"kubernetes.io/projected/8b398c60-0b41-4ecc-80d3-2e336559a338-kube-api-access-frt6v\") pod \"maas-api-revoke-sub-7f6cce00-hhd2g\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.135388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.135346    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:10.440349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.440322    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"]
Oct 30 13:16:10.442426 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:10.442398    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b398c60_0b41_4ecc_80d3_2e336559a338.slice/crio-a3dbde81e222b4d89072624b5169e39e521d694fb76ded63b8081c74ec7217d9 WatchSource:0}: Error finding container a3dbde81e222b4d89072624b5169e39e521d694fb76ded63b8081c74ec7217d9: Status 404 returned error can't find the container with id a3dbde81e222b4d89072624b5169e39e521d694fb76ded63b8081c74ec7217d9
Oct 30 13:16:10.822916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.822880    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g" event={"ID":"8b398c60-0b41-4ecc-80d3-2e336559a338","Type":"ContainerStarted","Data":"4cb7758f63b3d36d7899ac1ec5f093bb141850ee22e7c085e853afb7151b2342"}
Oct 30 13:16:10.823338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.822945    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g" event={"ID":"8b398c60-0b41-4ecc-80d3-2e336559a338","Type":"ContainerStarted","Data":"a3dbde81e222b4d89072624b5169e39e521d694fb76ded63b8081c74ec7217d9"}
Oct 30 13:16:10.864971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:10.864915    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g" podStartSLOduration=1.86490026 podStartE2EDuration="1.86490026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:10.862533162 +0000 UTC m=+781.122771423" watchObservedRunningTime="2026-10-30 13:16:10.86490026 +0000 UTC m=+781.125138494"
Oct 30 13:16:11.831372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:11.831334    4291 generic.go:356] "Generic (PLEG): container finished" podID="8b398c60-0b41-4ecc-80d3-2e336559a338" containerID="4cb7758f63b3d36d7899ac1ec5f093bb141850ee22e7c085e853afb7151b2342" exitCode=0
Oct 30 13:16:11.831889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:11.831384    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g" event={"ID":"8b398c60-0b41-4ecc-80d3-2e336559a338","Type":"ContainerDied","Data":"4cb7758f63b3d36d7899ac1ec5f093bb141850ee22e7c085e853afb7151b2342"}
Oct 30 13:16:12.984415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:12.984383    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:13.067171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.067140    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b398c60-0b41-4ecc-80d3-2e336559a338-kube-api-access-frt6v\" (UniqueName: \"kubernetes.io/projected/8b398c60-0b41-4ecc-80d3-2e336559a338-kube-api-access-frt6v\") pod \"8b398c60-0b41-4ecc-80d3-2e336559a338\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") "
Oct 30 13:16:13.067361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.067184    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-openshift-ca\") pod \"8b398c60-0b41-4ecc-80d3-2e336559a338\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") "
Oct 30 13:16:13.067361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.067256    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-xks-ca\") pod \"8b398c60-0b41-4ecc-80d3-2e336559a338\" (UID: \"8b398c60-0b41-4ecc-80d3-2e336559a338\") "
Oct 30 13:16:13.067604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.067562    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-openshift-ca" pod "8b398c60-0b41-4ecc-80d3-2e336559a338" (UID: "8b398c60-0b41-4ecc-80d3-2e336559a338"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:13.067711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.067688    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:13.069408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.069382    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-xks-ca" pod "8b398c60-0b41-4ecc-80d3-2e336559a338" (UID: "8b398c60-0b41-4ecc-80d3-2e336559a338"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:13.069673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.069620    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b398c60-0b41-4ecc-80d3-2e336559a338-kube-api-access-frt6v" pod "8b398c60-0b41-4ecc-80d3-2e336559a338" (UID: "8b398c60-0b41-4ecc-80d3-2e336559a338"). InnerVolumeSpecName "kube-api-access-frt6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:13.168446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.168408    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b398c60-0b41-4ecc-80d3-2e336559a338-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:13.168446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.168450    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frt6v\" (UniqueName: \"kubernetes.io/projected/8b398c60-0b41-4ecc-80d3-2e336559a338-kube-api-access-frt6v\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:13.847758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.847725    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"
Oct 30 13:16:13.847936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.847724    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g" event={"ID":"8b398c60-0b41-4ecc-80d3-2e336559a338","Type":"ContainerDied","Data":"a3dbde81e222b4d89072624b5169e39e521d694fb76ded63b8081c74ec7217d9"}
Oct 30 13:16:13.847936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:13.847849    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3dbde81e222b4d89072624b5169e39e521d694fb76ded63b8081c74ec7217d9"
Oct 30 13:16:15.598274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.598238    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"]
Oct 30 13:16:15.598745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.598732    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8b398c60-0b41-4ecc-80d3-2e336559a338" containerName="revoke-keys"
Oct 30 13:16:15.598789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.598749    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b398c60-0b41-4ecc-80d3-2e336559a338" containerName="revoke-keys"
Oct 30 13:16:15.598851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.598838    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b398c60-0b41-4ecc-80d3-2e336559a338" containerName="revoke-keys"
Oct 30 13:16:15.602103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.602079    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.619734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.619705    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"]
Oct 30 13:16:15.629192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.629167    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-f9ebb5\"/\"distinct-w0-f9ebb5-kserve-self-signed-certs\""
Oct 30 13:16:15.693736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.693694    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt9nd\" (UniqueName: \"kubernetes.io/projected/c8c75286-fcfd-4d26-9e75-de16b42a094c-kube-api-access-pt9nd\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.693981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.693752    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-home\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.693981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.693770    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-dshm\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.694122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.693996    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-model-cache\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.694122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.694089    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c75286-fcfd-4d26-9e75-de16b42a094c-tls-certs\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.694221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.694126    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-tmp-dir\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.794961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.794916    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt9nd\" (UniqueName: \"kubernetes.io/projected/c8c75286-fcfd-4d26-9e75-de16b42a094c-kube-api-access-pt9nd\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.794961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.794960    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-home\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.795234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.795157    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-dshm\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.795297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.795266    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-model-cache\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.795366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.795349    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c75286-fcfd-4d26-9e75-de16b42a094c-tls-certs\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.795431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.795387    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-tmp-dir\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.795485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.795446    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-home\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.795668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.795627    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-model-cache\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.795818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.795796    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-tmp-dir\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.799669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.798136    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c75286-fcfd-4d26-9e75-de16b42a094c-tls-certs\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.808760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.808728    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-dshm\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.818036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.818002    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt9nd\" (UniqueName: \"kubernetes.io/projected/c8c75286-fcfd-4d26-9e75-de16b42a094c-kube-api-access-pt9nd\") pod \"distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") " pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:15.914017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:15.913913    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:16.222334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:16.222307    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"]
Oct 30 13:16:16.224287 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:16.224253    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8c75286_fcfd_4d26_9e75_de16b42a094c.slice/crio-5959f505adcbb508057a450e2b6469183d8e8733507f5d207d69df6e3f950677 WatchSource:0}: Error finding container 5959f505adcbb508057a450e2b6469183d8e8733507f5d207d69df6e3f950677: Status 404 returned error can't find the container with id 5959f505adcbb508057a450e2b6469183d8e8733507f5d207d69df6e3f950677
Oct 30 13:16:16.868992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:16.868953    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" event={"ID":"c8c75286-fcfd-4d26-9e75-de16b42a094c","Type":"ContainerStarted","Data":"8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3"}
Oct 30 13:16:16.868992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:16.868995    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" event={"ID":"c8c75286-fcfd-4d26-9e75-de16b42a094c","Type":"ContainerStarted","Data":"5959f505adcbb508057a450e2b6469183d8e8733507f5d207d69df6e3f950677"}
Oct 30 13:16:16.914491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:16.914457    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:16.965438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:16.965375    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podStartSLOduration=1.96535969 podStartE2EDuration="1.96535969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:16.960486891 +0000 UTC m=+787.220725136" watchObservedRunningTime="2026-10-30 13:16:16.96535969 +0000 UTC m=+787.225597925"
Oct 30 13:16:17.965053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:17.965017    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:16:18.240628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.240547    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"]
Oct 30 13:16:18.244168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.244148    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.273927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.273876    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:16:18.274346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.274307    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:16:18.274346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.274329    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:16:18.274346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.274317    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:16:18.276383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.276362    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"]
Oct 30 13:16:18.421556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.421515    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.421793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.421580    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.421793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.421718    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc28n\" (UniqueName: \"kubernetes.io/projected/63c9f3c4-b806-48aa-8e30-c64719a98c06-kube-api-access-rc28n\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.522947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.522849    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc28n\" (UniqueName: \"kubernetes.io/projected/63c9f3c4-b806-48aa-8e30-c64719a98c06-kube-api-access-rc28n\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.523117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.522971    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.523117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.523023    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.523696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.523675    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.525500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.525473    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.543808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.543777    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc28n\" (UniqueName: \"kubernetes.io/projected/63c9f3c4-b806-48aa-8e30-c64719a98c06-kube-api-access-rc28n\") pod \"maas-api-revoke-sub-555de3ea-275bv\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.557477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.557453    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:18.885777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:18.885752    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"]
Oct 30 13:16:19.887105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:19.887069    4291 generic.go:356] "Generic (PLEG): container finished" podID="63c9f3c4-b806-48aa-8e30-c64719a98c06" containerID="598d5dd1fc843e33b6bbc90ad91a93fc2e31df37c4c0c6716ce8ffe67a3696a4" exitCode=0
Oct 30 13:16:19.887497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:19.887153    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv" event={"ID":"63c9f3c4-b806-48aa-8e30-c64719a98c06","Type":"ContainerDied","Data":"598d5dd1fc843e33b6bbc90ad91a93fc2e31df37c4c0c6716ce8ffe67a3696a4"}
Oct 30 13:16:19.887497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:19.887202    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv" event={"ID":"63c9f3c4-b806-48aa-8e30-c64719a98c06","Type":"ContainerStarted","Data":"2d4411eed015fe2691288416aa5eb7664ab9958bea4b33177a0e0f9cdd225cb2"}
Oct 30 13:16:20.875518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:20.875483    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"]
Oct 30 13:16:20.879053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:20.879032    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:20.901598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:20.901558    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"]
Oct 30 13:16:21.048121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048083    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048142    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048266    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048317    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048363    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048410    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048479    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048518    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wb59\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-kube-api-access-5wb59\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.048707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.048593    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.065658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.062348    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:21.149410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149315    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63c9f3c4-b806-48aa-8e30-c64719a98c06-kube-api-access-rc28n\" (UniqueName: \"kubernetes.io/projected/63c9f3c4-b806-48aa-8e30-c64719a98c06-kube-api-access-rc28n\") pod \"63c9f3c4-b806-48aa-8e30-c64719a98c06\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") "
Oct 30 13:16:21.149559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149477    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-xks-ca\") pod \"63c9f3c4-b806-48aa-8e30-c64719a98c06\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") "
Oct 30 13:16:21.149559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149548    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-openshift-ca\") pod \"63c9f3c4-b806-48aa-8e30-c64719a98c06\" (UID: \"63c9f3c4-b806-48aa-8e30-c64719a98c06\") "
Oct 30 13:16:21.149719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149698    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.149766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149751    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wb59\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-kube-api-access-5wb59\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149825    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149874    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149921    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149967    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.149978    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-openshift-ca" pod "63c9f3c4-b806-48aa-8e30-c64719a98c06" (UID: "63c9f3c4-b806-48aa-8e30-c64719a98c06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150005    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150050    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150097    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150185    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150228    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.150352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150260    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.151111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150528    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.151111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150616    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.151111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.150892    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.151900 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.151866    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-xks-ca" pod "63c9f3c4-b806-48aa-8e30-c64719a98c06" (UID: "63c9f3c4-b806-48aa-8e30-c64719a98c06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:21.152132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.152108    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.152595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.152563    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.152679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.152583    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63c9f3c4-b806-48aa-8e30-c64719a98c06-kube-api-access-rc28n" pod "63c9f3c4-b806-48aa-8e30-c64719a98c06" (UID: "63c9f3c4-b806-48aa-8e30-c64719a98c06"). InnerVolumeSpecName "kube-api-access-rc28n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:21.170444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.170410    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.170591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.170480    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wb59\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-kube-api-access-5wb59\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.194902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.194532    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:21.251674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.251603    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63c9f3c4-b806-48aa-8e30-c64719a98c06-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:21.251846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.251727    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc28n\" (UniqueName: \"kubernetes.io/projected/63c9f3c4-b806-48aa-8e30-c64719a98c06-kube-api-access-rc28n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:21.525147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.524537    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"]
Oct 30 13:16:21.543249 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:21.542922    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode73a9c61_a0fd_4802_9864_cf2d8bde666b.slice/crio-38d3ebd4a71fe180c1d4fc45bce886a1aa955ff7ec5a0fd1d90e3378216d97cf WatchSource:0}: Error finding container 38d3ebd4a71fe180c1d4fc45bce886a1aa955ff7ec5a0fd1d90e3378216d97cf: Status 404 returned error can't find the container with id 38d3ebd4a71fe180c1d4fc45bce886a1aa955ff7ec5a0fd1d90e3378216d97cf
Oct 30 13:16:21.545156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.545117    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:21.545290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.545207    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:21.545290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.545245    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:21.905449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.905407    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" event={"ID":"e73a9c61-a0fd-4802-9864-cf2d8bde666b","Type":"ContainerStarted","Data":"1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c"}
Oct 30 13:16:21.905891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.905475    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" event={"ID":"e73a9c61-a0fd-4802-9864-cf2d8bde666b","Type":"ContainerStarted","Data":"38d3ebd4a71fe180c1d4fc45bce886a1aa955ff7ec5a0fd1d90e3378216d97cf"}
Oct 30 13:16:21.907028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.907002    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv" event={"ID":"63c9f3c4-b806-48aa-8e30-c64719a98c06","Type":"ContainerDied","Data":"2d4411eed015fe2691288416aa5eb7664ab9958bea4b33177a0e0f9cdd225cb2"}
Oct 30 13:16:21.907151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.907054    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d4411eed015fe2691288416aa5eb7664ab9958bea4b33177a0e0f9cdd225cb2"
Oct 30 13:16:21.907151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.907009    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"
Oct 30 13:16:21.946426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:21.946357    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podStartSLOduration=1.9463359169999999 podStartE2EDuration="1.946335917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:21.945290762 +0000 UTC m=+792.205528995" watchObservedRunningTime="2026-10-30 13:16:21.946335917 +0000 UTC m=+792.206574152"
Oct 30 13:16:22.195562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:22.195465    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:22.197158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:22.197130    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 13:16:22.197269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:22.197181    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 13:16:23.195057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:23.195021    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 13:16:23.195580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:23.195087    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 13:16:24.195522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:24.195486    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 13:16:24.195958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:24.195538    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 13:16:25.195485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:25.195449    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 13:16:25.195697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:25.195507    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 13:16:25.914734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:25.914694    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:25.928078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:25.928052    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:25.945201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:25.945168    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:16:26.195245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:26.195154    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 13:16:26.195245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:26.195213    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 13:16:27.195548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:27.195504    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 13:16:27.195953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:27.195585    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 13:16:28.195270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:28.195232    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused" start-of-body=
Oct 30 13:16:28.195440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:28.195313    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": dial tcp 10.133.0.62:15021: connect: connection refused"
Oct 30 13:16:30.236826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:30.236776    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:16:30.237257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:30.236861    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:16:30.406608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:30.406572    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:30.407118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:30.407097    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:30.407838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:30.407819    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 13:16:33.230322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.230286    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"]
Oct 30 13:16:33.230851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.230834    4291 cpu_manager.go:414] "containerMap: removing container" podUID="63c9f3c4-b806-48aa-8e30-c64719a98c06" containerName="revoke-keys"
Oct 30 13:16:33.230903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.230852    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63c9f3c4-b806-48aa-8e30-c64719a98c06" containerName="revoke-keys"
Oct 30 13:16:33.230949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.230934    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="63c9f3c4-b806-48aa-8e30-c64719a98c06" containerName="revoke-keys"
Oct 30 13:16:33.234290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.234275    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.250555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.250530    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"]
Oct 30 13:16:33.260032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.260010    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:16:33.260159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.260097    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:16:33.260159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.260125    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:16:33.261446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.261425    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:16:33.273093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.273069    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.273224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.273140    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.273224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.273163    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtk9l\" (UniqueName: \"kubernetes.io/projected/7d71bfbe-492a-40be-a378-f80a80d22772-kube-api-access-wtk9l\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.373884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.373845    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.374064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.373891    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtk9l\" (UniqueName: \"kubernetes.io/projected/7d71bfbe-492a-40be-a378-f80a80d22772-kube-api-access-wtk9l\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.374064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.374007    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.374747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.374725    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.376395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.376377    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.392962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.392936    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtk9l\" (UniqueName: \"kubernetes.io/projected/7d71bfbe-492a-40be-a378-f80a80d22772-kube-api-access-wtk9l\") pod \"maas-api-revoke-sub-24428ba5-ltj5m\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.547860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.547778    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:33.848227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.848203    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"]
Oct 30 13:16:33.850596 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:33.850565    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d71bfbe_492a_40be_a378_f80a80d22772.slice/crio-0cba042e28c2aa5bef93e29028249453fa6bc9baac678dde5dd422fc1f9d4ebf WatchSource:0}: Error finding container 0cba042e28c2aa5bef93e29028249453fa6bc9baac678dde5dd422fc1f9d4ebf: Status 404 returned error can't find the container with id 0cba042e28c2aa5bef93e29028249453fa6bc9baac678dde5dd422fc1f9d4ebf
Oct 30 13:16:33.976875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:33.976837    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m" event={"ID":"7d71bfbe-492a-40be-a378-f80a80d22772","Type":"ContainerStarted","Data":"0cba042e28c2aa5bef93e29028249453fa6bc9baac678dde5dd422fc1f9d4ebf"}
Oct 30 13:16:34.985167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:34.985074    4291 generic.go:356] "Generic (PLEG): container finished" podID="7d71bfbe-492a-40be-a378-f80a80d22772" containerID="84322b3bc02ad61d0717a5383fba08a143e5b257095aa2f7baa10c32533ddb62" exitCode=0
Oct 30 13:16:34.985604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:34.985157    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m" event={"ID":"7d71bfbe-492a-40be-a378-f80a80d22772","Type":"ContainerDied","Data":"84322b3bc02ad61d0717a5383fba08a143e5b257095aa2f7baa10c32533ddb62"}
Oct 30 13:16:35.931865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:35.931833    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"]
Oct 30 13:16:35.935664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:35.935621    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:35.951357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:35.951320    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:16:35.995247 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:35.995218    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"]
Oct 30 13:16:36.002887 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.002858    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.002932    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.003065    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.003140    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.003198    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.003249    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b42n\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-kube-api-access-9b42n\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.003399    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.003481    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.003540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.003516    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.104490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104454    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.104735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104497    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.104735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104538    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.104735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104581    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.104735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104621    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.104735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104664    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.104735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104714    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.105047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104760    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b42n\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-kube-api-access-9b42n\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.105047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.104840    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.105047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.105009    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-certs\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.105224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.105083    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.105278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.105251    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-credential-socket\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.105355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.105327    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-data\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.105406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.105339    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istiod-ca-cert\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.107102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.107072    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-envoy\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.107197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.107110    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-podinfo\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.131692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.131671    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-token\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.134117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.134101    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:36.136772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.136754    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b42n\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-kube-api-access-9b42n\") pod \"e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") " pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.205304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.205239    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-xks-ca\") pod \"7d71bfbe-492a-40be-a378-f80a80d22772\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") "
Oct 30 13:16:36.205415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.205375    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d71bfbe-492a-40be-a378-f80a80d22772-kube-api-access-wtk9l\" (UniqueName: \"kubernetes.io/projected/7d71bfbe-492a-40be-a378-f80a80d22772-kube-api-access-wtk9l\") pod \"7d71bfbe-492a-40be-a378-f80a80d22772\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") "
Oct 30 13:16:36.205475 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.205458    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-openshift-ca\") pod \"7d71bfbe-492a-40be-a378-f80a80d22772\" (UID: \"7d71bfbe-492a-40be-a378-f80a80d22772\") "
Oct 30 13:16:36.205918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.205886    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-openshift-ca" pod "7d71bfbe-492a-40be-a378-f80a80d22772" (UID: "7d71bfbe-492a-40be-a378-f80a80d22772"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:36.207803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.207777    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d71bfbe-492a-40be-a378-f80a80d22772-kube-api-access-wtk9l" pod "7d71bfbe-492a-40be-a378-f80a80d22772" (UID: "7d71bfbe-492a-40be-a378-f80a80d22772"). InnerVolumeSpecName "kube-api-access-wtk9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:36.208073 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.208051    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-xks-ca" pod "7d71bfbe-492a-40be-a378-f80a80d22772" (UID: "7d71bfbe-492a-40be-a378-f80a80d22772"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:36.255325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.255285    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:36.307479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.307445    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.307563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.307487    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtk9l\" (UniqueName: \"kubernetes.io/projected/7d71bfbe-492a-40be-a378-f80a80d22772-kube-api-access-wtk9l\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.307563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.307503    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d71bfbe-492a-40be-a378-f80a80d22772-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:36.580813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.580785    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"]
Oct 30 13:16:36.583936 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:36.583904    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452f6b54_cb09_463e_acaa_0b4d046ac9a1.slice/crio-dfb8ff54d85f69c88216514ee97249e6badaf6a8f8c4520bf477ecbe672fb22c WatchSource:0}: Error finding container dfb8ff54d85f69c88216514ee97249e6badaf6a8f8c4520bf477ecbe672fb22c: Status 404 returned error can't find the container with id dfb8ff54d85f69c88216514ee97249e6badaf6a8f8c4520bf477ecbe672fb22c
Oct 30 13:16:36.586578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.586538    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:36.586714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.586629    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:36.586714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:36.586700    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:16:37.039661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.031032    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m" event={"ID":"7d71bfbe-492a-40be-a378-f80a80d22772","Type":"ContainerDied","Data":"0cba042e28c2aa5bef93e29028249453fa6bc9baac678dde5dd422fc1f9d4ebf"}
Oct 30 13:16:37.039661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.031096    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cba042e28c2aa5bef93e29028249453fa6bc9baac678dde5dd422fc1f9d4ebf"
Oct 30 13:16:37.039661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.032387    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"
Oct 30 13:16:37.039661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.037039    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" event={"ID":"452f6b54-cb09-463e-acaa-0b4d046ac9a1","Type":"ContainerStarted","Data":"1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745"}
Oct 30 13:16:37.039661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.037092    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" event={"ID":"452f6b54-cb09-463e-acaa-0b4d046ac9a1","Type":"ContainerStarted","Data":"dfb8ff54d85f69c88216514ee97249e6badaf6a8f8c4520bf477ecbe672fb22c"}
Oct 30 13:16:37.100664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.088666    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podStartSLOduration=2.088628253 podStartE2EDuration="2.088628253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:37.085880575 +0000 UTC m=+807.346118867" watchObservedRunningTime="2026-10-30 13:16:37.088628253 +0000 UTC m=+807.348866492"
Oct 30 13:16:37.261586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.261549    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:37.261947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.261927    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:37.262113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:37.262090    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:38.262689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:38.262427    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:38.262689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:38.262506    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:38.379263 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:16:38.379221    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:16:39.258033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:39.257994    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:39.258358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:39.258317    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:40.256010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:40.255972    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:40.256577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:40.256545    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:41.257011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:41.256973    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:41.257619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:41.257583    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:42.269660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:42.268064    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:42.269660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:42.268143    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:43.270172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:43.270130    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:43.270772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:43.270738    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:44.256013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:44.255975    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused" start-of-body=
Oct 30 13:16:44.256260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:44.256027    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.64:15021/healthz/ready\": dial tcp 10.133.0.64:15021: connect: connection refused"
Oct 30 13:16:45.260121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:45.260092    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:45.260521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:45.260482    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:45.261051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:45.261036    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:16:46.117768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.117737    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"]
Oct 30 13:16:46.118132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.118119    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7d71bfbe-492a-40be-a378-f80a80d22772" containerName="revoke-keys"
Oct 30 13:16:46.118179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.118133    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d71bfbe-492a-40be-a378-f80a80d22772" containerName="revoke-keys"
Oct 30 13:16:46.118250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.118241    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d71bfbe-492a-40be-a378-f80a80d22772" containerName="revoke-keys"
Oct 30 13:16:46.121585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.121565    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.136693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.136662    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/54cd6743-b67e-4236-a62b-2005ba1b96b7-kube-api-access-6pzfq\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.136917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.136898    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.137001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.136949    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.137349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.137325    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:16:46.139282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.139254    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"]
Oct 30 13:16:46.147705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.147687    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:16:46.147705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.147693    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:16:46.147934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.147921    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:16:46.238360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.238326    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.238360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.238362    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/54cd6743-b67e-4236-a62b-2005ba1b96b7-kube-api-access-6pzfq\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.238626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.238520    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.239274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.239250    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.240854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.240836    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.257965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.257938    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/54cd6743-b67e-4236-a62b-2005ba1b96b7-kube-api-access-6pzfq\") pod \"maas-api-revoke-sub-44dfaa8b-ckpb2\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.435058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.434969    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:46.742438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:46.742361    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"]
Oct 30 13:16:46.746237 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:46.746206    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54cd6743_b67e_4236_a62b_2005ba1b96b7.slice/crio-881af1a0a43abd0c9545377572ec1c8809c7e2d30cbb0a66a8140579c43c36dc WatchSource:0}: Error finding container 881af1a0a43abd0c9545377572ec1c8809c7e2d30cbb0a66a8140579c43c36dc: Status 404 returned error can't find the container with id 881af1a0a43abd0c9545377572ec1c8809c7e2d30cbb0a66a8140579c43c36dc
Oct 30 13:16:47.200939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:47.200904    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2" event={"ID":"54cd6743-b67e-4236-a62b-2005ba1b96b7","Type":"ContainerStarted","Data":"1de14dfbed712a92dfe6582eb5885fde04598f5117bb8e3316411c6f24e58c0e"}
Oct 30 13:16:47.201120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:47.200963    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2" event={"ID":"54cd6743-b67e-4236-a62b-2005ba1b96b7","Type":"ContainerStarted","Data":"881af1a0a43abd0c9545377572ec1c8809c7e2d30cbb0a66a8140579c43c36dc"}
Oct 30 13:16:47.241335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:47.241278    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2" podStartSLOduration=1.241265111 podStartE2EDuration="1.241265111s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:47.239568214 +0000 UTC m=+817.499806475" watchObservedRunningTime="2026-10-30 13:16:47.241265111 +0000 UTC m=+817.501503345"
Oct 30 13:16:48.208534 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:48.208497    4291 generic.go:356] "Generic (PLEG): container finished" podID="54cd6743-b67e-4236-a62b-2005ba1b96b7" containerID="1de14dfbed712a92dfe6582eb5885fde04598f5117bb8e3316411c6f24e58c0e" exitCode=0
Oct 30 13:16:48.208934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:48.208558    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2" event={"ID":"54cd6743-b67e-4236-a62b-2005ba1b96b7","Type":"ContainerDied","Data":"1de14dfbed712a92dfe6582eb5885fde04598f5117bb8e3316411c6f24e58c0e"}
Oct 30 13:16:49.355255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.355231    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:49.473900 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.473827    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54cd6743-b67e-4236-a62b-2005ba1b96b7-kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/54cd6743-b67e-4236-a62b-2005ba1b96b7-kube-api-access-6pzfq\") pod \"54cd6743-b67e-4236-a62b-2005ba1b96b7\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") "
Oct 30 13:16:49.474065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.473907    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-xks-ca\") pod \"54cd6743-b67e-4236-a62b-2005ba1b96b7\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") "
Oct 30 13:16:49.474065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.474000    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-openshift-ca\") pod \"54cd6743-b67e-4236-a62b-2005ba1b96b7\" (UID: \"54cd6743-b67e-4236-a62b-2005ba1b96b7\") "
Oct 30 13:16:49.474396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.474367    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-openshift-ca" pod "54cd6743-b67e-4236-a62b-2005ba1b96b7" (UID: "54cd6743-b67e-4236-a62b-2005ba1b96b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:16:49.475936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.475904    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-xks-ca" pod "54cd6743-b67e-4236-a62b-2005ba1b96b7" (UID: "54cd6743-b67e-4236-a62b-2005ba1b96b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:16:49.476032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.475978    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54cd6743-b67e-4236-a62b-2005ba1b96b7-kube-api-access-6pzfq" pod "54cd6743-b67e-4236-a62b-2005ba1b96b7" (UID: "54cd6743-b67e-4236-a62b-2005ba1b96b7"). InnerVolumeSpecName "kube-api-access-6pzfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:16:49.575332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.575299    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pzfq\" (UniqueName: \"kubernetes.io/projected/54cd6743-b67e-4236-a62b-2005ba1b96b7-kube-api-access-6pzfq\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:49.575332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.575327    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:49.575332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:49.575336    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54cd6743-b67e-4236-a62b-2005ba1b96b7-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:16:50.221322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.221238    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"
Oct 30 13:16:50.221322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.221272    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2" event={"ID":"54cd6743-b67e-4236-a62b-2005ba1b96b7","Type":"ContainerDied","Data":"881af1a0a43abd0c9545377572ec1c8809c7e2d30cbb0a66a8140579c43c36dc"}
Oct 30 13:16:50.221514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.221327    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881af1a0a43abd0c9545377572ec1c8809c7e2d30cbb0a66a8140579c43c36dc"
Oct 30 13:16:50.960263 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.960227    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"]
Oct 30 13:16:50.960896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.960875    4291 cpu_manager.go:414] "containerMap: removing container" podUID="54cd6743-b67e-4236-a62b-2005ba1b96b7" containerName="revoke-keys"
Oct 30 13:16:50.960962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.960899    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54cd6743-b67e-4236-a62b-2005ba1b96b7" containerName="revoke-keys"
Oct 30 13:16:50.961064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.961051    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="54cd6743-b67e-4236-a62b-2005ba1b96b7" containerName="revoke-keys"
Oct 30 13:16:50.964821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.964794    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:50.980712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.980683    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"]
Oct 30 13:16:50.987867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:50.987844    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-52af49\"/\"distinct-2-w1-52af49-kserve-self-signed-certs\""
Oct 30 13:16:51.091744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.091700    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b8m4\" (UniqueName: \"kubernetes.io/projected/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-kube-api-access-2b8m4\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.091926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.091841    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tmp-dir\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.091926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.091895    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tls-certs\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.091926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.091920    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-dshm\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.092036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.092017    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-home\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.092071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.092051    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-model-cache\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193071    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b8m4\" (UniqueName: \"kubernetes.io/projected/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-kube-api-access-2b8m4\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193137    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tmp-dir\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193161    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tls-certs\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193189    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-dshm\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193216    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-home\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193239    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-model-cache\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193582    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tmp-dir\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193762    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-home\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.193897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.193719    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-model-cache\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.195610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.195583    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-dshm\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.195881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.195860    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tls-certs\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.212074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.211998    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b8m4\" (UniqueName: \"kubernetes.io/projected/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-kube-api-access-2b8m4\") pod \"distinct-2-w1-52af49-kserve-55bd98c68-9q2jh\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") " pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.280296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.280254    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:51.586914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:51.586886    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"]
Oct 30 13:16:51.588585 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:51.588554    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod07d433e1_a2e5_46bf_9dbf_e8e946980a1b.slice/crio-52e834e094f76421132a452c919a540ee80bd2dd186a5af573e212a724c1363d WatchSource:0}: Error finding container 52e834e094f76421132a452c919a540ee80bd2dd186a5af573e212a724c1363d: Status 404 returned error can't find the container with id 52e834e094f76421132a452c919a540ee80bd2dd186a5af573e212a724c1363d
Oct 30 13:16:52.238324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:52.238285    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" event={"ID":"07d433e1-a2e5-46bf-9dbf-e8e946980a1b","Type":"ContainerStarted","Data":"289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723"}
Oct 30 13:16:52.238875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:52.238334    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" event={"ID":"07d433e1-a2e5-46bf-9dbf-e8e946980a1b","Type":"ContainerStarted","Data":"52e834e094f76421132a452c919a540ee80bd2dd186a5af573e212a724c1363d"}
Oct 30 13:16:52.278862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:52.278805    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podStartSLOduration=2.2787907880000002 podStartE2EDuration="2.278790788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:16:52.276662119 +0000 UTC m=+822.536900347" watchObservedRunningTime="2026-10-30 13:16:52.278790788 +0000 UTC m=+822.539029022"
Oct 30 13:16:52.280694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:52.280670    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:16:59.373840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.373801    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"]
Oct 30 13:16:59.376410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.376392    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.394824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.394801    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"]
Oct 30 13:16:59.403735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.403712    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:16:59.403866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.403810    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:16:59.403866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.403812    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:16:59.403974 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.403881    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:16:59.488601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.488571    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq8nm\" (UniqueName: \"kubernetes.io/projected/6498d357-c8a3-4943-9437-3c8dae7bcea6-kube-api-access-jq8nm\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.488775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.488747    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.488818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.488778    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.590048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.590004    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq8nm\" (UniqueName: \"kubernetes.io/projected/6498d357-c8a3-4943-9437-3c8dae7bcea6-kube-api-access-jq8nm\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.590220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.590165    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.590220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.590205    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.590851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.590827    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.592483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.592462    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.608621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.608596    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq8nm\" (UniqueName: \"kubernetes.io/projected/6498d357-c8a3-4943-9437-3c8dae7bcea6-kube-api-access-jq8nm\") pod \"maas-api-revoke-sub-a2ec11de-qv8v4\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.688472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.688396    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:16:59.989277 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:16:59.989243    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"]
Oct 30 13:16:59.990753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:16:59.990724    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6498d357_c8a3_4943_9437_3c8dae7bcea6.slice/crio-d18c8dadbc404a0856eaaa7a9d4e142c596fbc3328b66a030f794cec2b280b45 WatchSource:0}: Error finding container d18c8dadbc404a0856eaaa7a9d4e142c596fbc3328b66a030f794cec2b280b45: Status 404 returned error can't find the container with id d18c8dadbc404a0856eaaa7a9d4e142c596fbc3328b66a030f794cec2b280b45
Oct 30 13:17:00.288390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:00.288357    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4" event={"ID":"6498d357-c8a3-4943-9437-3c8dae7bcea6","Type":"ContainerStarted","Data":"476652d7034ec1c27eddd46c54e0cbc880492341a27adaadaf9333c903f6090e"}
Oct 30 13:17:00.288581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:00.288409    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4" event={"ID":"6498d357-c8a3-4943-9437-3c8dae7bcea6","Type":"ContainerStarted","Data":"d18c8dadbc404a0856eaaa7a9d4e142c596fbc3328b66a030f794cec2b280b45"}
Oct 30 13:17:00.326664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:00.326585    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4" podStartSLOduration=1.326571513 podStartE2EDuration="1.326571513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:00.325475113 +0000 UTC m=+830.585713346" watchObservedRunningTime="2026-10-30 13:17:00.326571513 +0000 UTC m=+830.586809772"
Oct 30 13:17:01.280584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:01.280486    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:17:01.293404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:01.293378    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:17:01.296150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:01.296124    4291 generic.go:356] "Generic (PLEG): container finished" podID="6498d357-c8a3-4943-9437-3c8dae7bcea6" containerID="476652d7034ec1c27eddd46c54e0cbc880492341a27adaadaf9333c903f6090e" exitCode=0
Oct 30 13:17:01.296270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:01.296211    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4" event={"ID":"6498d357-c8a3-4943-9437-3c8dae7bcea6","Type":"ContainerDied","Data":"476652d7034ec1c27eddd46c54e0cbc880492341a27adaadaf9333c903f6090e"}
Oct 30 13:17:01.307295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:01.307273    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:17:02.442391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.442364    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:17:02.525031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.524988    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-xks-ca\") pod \"6498d357-c8a3-4943-9437-3c8dae7bcea6\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") "
Oct 30 13:17:02.525233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.525149    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-openshift-ca\") pod \"6498d357-c8a3-4943-9437-3c8dae7bcea6\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") "
Oct 30 13:17:02.525233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.525194    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6498d357-c8a3-4943-9437-3c8dae7bcea6-kube-api-access-jq8nm\" (UniqueName: \"kubernetes.io/projected/6498d357-c8a3-4943-9437-3c8dae7bcea6-kube-api-access-jq8nm\") pod \"6498d357-c8a3-4943-9437-3c8dae7bcea6\" (UID: \"6498d357-c8a3-4943-9437-3c8dae7bcea6\") "
Oct 30 13:17:02.525532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.525502    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-openshift-ca" pod "6498d357-c8a3-4943-9437-3c8dae7bcea6" (UID: "6498d357-c8a3-4943-9437-3c8dae7bcea6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:02.525715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.525697    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:02.527239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.527214    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-xks-ca" pod "6498d357-c8a3-4943-9437-3c8dae7bcea6" (UID: "6498d357-c8a3-4943-9437-3c8dae7bcea6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:02.527325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.527311    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6498d357-c8a3-4943-9437-3c8dae7bcea6-kube-api-access-jq8nm" pod "6498d357-c8a3-4943-9437-3c8dae7bcea6" (UID: "6498d357-c8a3-4943-9437-3c8dae7bcea6"). InnerVolumeSpecName "kube-api-access-jq8nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:02.626305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.626273    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6498d357-c8a3-4943-9437-3c8dae7bcea6-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:02.626305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:02.626302    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq8nm\" (UniqueName: \"kubernetes.io/projected/6498d357-c8a3-4943-9437-3c8dae7bcea6-kube-api-access-jq8nm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:03.311428 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:03.311398    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"
Oct 30 13:17:03.311595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:03.311435    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4" event={"ID":"6498d357-c8a3-4943-9437-3c8dae7bcea6","Type":"ContainerDied","Data":"d18c8dadbc404a0856eaaa7a9d4e142c596fbc3328b66a030f794cec2b280b45"}
Oct 30 13:17:03.311595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:03.311495    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d18c8dadbc404a0856eaaa7a9d4e142c596fbc3328b66a030f794cec2b280b45"
Oct 30 13:17:05.445631 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.445593    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"]
Oct 30 13:17:05.446062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.446048    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6498d357-c8a3-4943-9437-3c8dae7bcea6" containerName="revoke-keys"
Oct 30 13:17:05.446104 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.446063    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6498d357-c8a3-4943-9437-3c8dae7bcea6" containerName="revoke-keys"
Oct 30 13:17:05.446150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.446141    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6498d357-c8a3-4943-9437-3c8dae7bcea6" containerName="revoke-keys"
Oct 30 13:17:05.454747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.452930    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.478227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.478194    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"]
Oct 30 13:17:05.486373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.486339    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-9bf8ad-openshift-default-dockercfg-shhh4\""
Oct 30 13:17:05.559621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.559570    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.559823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.559646    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed52cc6-2abf-47e5-979c-feadab32d548-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.559823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.559795    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.559912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.559839    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.559912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.559870    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.559978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.559940    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.560025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.560009    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed52cc6-2abf-47e5-979c-feadab32d548-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.560062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.560041    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpfg2\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-kube-api-access-gpfg2\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.560216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.560195    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661375    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661423    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661452    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661477    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661526    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed52cc6-2abf-47e5-979c-feadab32d548-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661550    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpfg2\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-kube-api-access-gpfg2\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661691    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661769    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661798    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed52cc6-2abf-47e5-979c-feadab32d548-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.661907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661899    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.662082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.661931    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.662115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.662096    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.662152 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.662119    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.662802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.662783    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed52cc6-2abf-47e5-979c-feadab32d548-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.663895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.663875    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.664021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.664005    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed52cc6-2abf-47e5-979c-feadab32d548-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.680401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.680376    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.680883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.680865    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpfg2\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-kube-api-access-gpfg2\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:05.786198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:05.786116    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:06.105222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:06.105188    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"]
Oct 30 13:17:06.106782 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:06.106754    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeed52cc6_2abf_47e5_979c_feadab32d548.slice/crio-4403007b9404f0c82754877687503f1193a381ea6701ba13416722bf47a4e868 WatchSource:0}: Error finding container 4403007b9404f0c82754877687503f1193a381ea6701ba13416722bf47a4e868: Status 404 returned error can't find the container with id 4403007b9404f0c82754877687503f1193a381ea6701ba13416722bf47a4e868
Oct 30 13:17:06.109020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:06.108990    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:06.109100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:06.109058    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:06.109141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:06.109103    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:06.332055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:06.332016    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" event={"ID":"eed52cc6-2abf-47e5-979c-feadab32d548","Type":"ContainerStarted","Data":"71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7"}
Oct 30 13:17:06.332273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:06.332081    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" event={"ID":"eed52cc6-2abf-47e5-979c-feadab32d548","Type":"ContainerStarted","Data":"4403007b9404f0c82754877687503f1193a381ea6701ba13416722bf47a4e868"}
Oct 30 13:17:06.787195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:06.787155    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:07.788357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:07.788315    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:17:07.788776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:07.788383    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:17:07.823432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:07.823382    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podStartSLOduration=2.823365688 podStartE2EDuration="2.823365688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:06.385097558 +0000 UTC m=+836.645335770" watchObservedRunningTime="2026-10-30 13:17:07.823365688 +0000 UTC m=+838.083603919"
Oct 30 13:17:07.826378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:07.826354    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:17:08.210035 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:17:08.209992    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:17:08.401105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.400975    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"]
Oct 30 13:17:08.405877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.405801    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.424364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.424334    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-ec41e5-openshift-default-dockercfg-6fkcb\""
Oct 30 13:17:08.448165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.447035    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"]
Oct 30 13:17:08.499448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.499342    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.499755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.499729    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.499920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.499903    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.500240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.500203    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.500388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.500371    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.500503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.500488    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.500690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.500674    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmbf\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-kube-api-access-kjmbf\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.500815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.500800    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.500919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.500905    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602276 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602226    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmbf\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-kube-api-access-kjmbf\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602281    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602316    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602350    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602398    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602457    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602573    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602617    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602663    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602833    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.602921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.602907    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.603078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.603058    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.603181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.603153    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.603666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.603601    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.605944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.605514    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.605944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.605885    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.626123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.626088    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmbf\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-kube-api-access-kjmbf\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.626411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.626374    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.722659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.720156    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:08.796553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.796066    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:17:08.796553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:08.796126    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:17:09.262678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.251248    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"]
Oct 30 13:17:09.337469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.337424    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:09.337740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.337716    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:09.341790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.341744    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:09.521724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.511717    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" event={"ID":"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44","Type":"ContainerStarted","Data":"c883c6de0994fd148ca2df6eda8bbaa8f3bd21f7580ecab4ecb448e6ef41731d"}
Oct 30 13:17:09.787853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.787262    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:17:09.787853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.787327    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:17:09.937886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.937840    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"]
Oct 30 13:17:09.950976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.950274    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.026690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.979976    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:10.026690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.995489    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:10.026690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.995584    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:17:10.026690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:09.995655    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:10.059879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.050796    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"]
Oct 30 13:17:10.152421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.152373    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.152627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.152477    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.152627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.152510    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whnmv\" (UniqueName: \"kubernetes.io/projected/dfdb1914-f8a5-49c6-bfbb-160bb380737b-kube-api-access-whnmv\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.264660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.261240    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.264660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.261299    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whnmv\" (UniqueName: \"kubernetes.io/projected/dfdb1914-f8a5-49c6-bfbb-160bb380737b-kube-api-access-whnmv\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.264660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.261460    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.264660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.262302    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.297542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.293459    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whnmv\" (UniqueName: \"kubernetes.io/projected/dfdb1914-f8a5-49c6-bfbb-160bb380737b-kube-api-access-whnmv\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.314129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.303451    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82d5fead-pltnm\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.446657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.446494    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:10.524678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.523933    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" event={"ID":"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44","Type":"ContainerStarted","Data":"4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb"}
Oct 30 13:17:10.606578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.606523    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podStartSLOduration=2.606474861 podStartE2EDuration="2.606474861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:10.593180046 +0000 UTC m=+840.853418307" watchObservedRunningTime="2026-10-30 13:17:10.606474861 +0000 UTC m=+840.866713094"
Oct 30 13:17:10.720502 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.720469    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:10.721826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.721800    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:17:10.721952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.721855    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:17:10.795553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.795092    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:17:10.795553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.795155    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:17:10.814945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:10.814917    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"]
Oct 30 13:17:10.816453 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:10.816419    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfdb1914_f8a5_49c6_bfbb_160bb380737b.slice/crio-9f9e71fec2aa0b3e51ac354e7c16c79c9d529af5e47aeed3de7eb017d2062f16 WatchSource:0}: Error finding container 9f9e71fec2aa0b3e51ac354e7c16c79c9d529af5e47aeed3de7eb017d2062f16: Status 404 returned error can't find the container with id 9f9e71fec2aa0b3e51ac354e7c16c79c9d529af5e47aeed3de7eb017d2062f16
Oct 30 13:17:11.533296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:11.533209    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm" event={"ID":"dfdb1914-f8a5-49c6-bfbb-160bb380737b","Type":"ContainerStarted","Data":"b815e3ccd9c401ae098e77ae1435d5e0039aefd28cdc4f7d647bfbdffffe9077"}
Oct 30 13:17:11.533296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:11.533263    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm" event={"ID":"dfdb1914-f8a5-49c6-bfbb-160bb380737b","Type":"ContainerStarted","Data":"9f9e71fec2aa0b3e51ac354e7c16c79c9d529af5e47aeed3de7eb017d2062f16"}
Oct 30 13:17:11.578112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:11.578024    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm" podStartSLOduration=2.578001175 podStartE2EDuration="2.578001175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:11.577929499 +0000 UTC m=+841.838167734" watchObservedRunningTime="2026-10-30 13:17:11.578001175 +0000 UTC m=+841.838239410"
Oct 30 13:17:11.720889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:11.720853    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:17:11.721118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:11.720919    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:17:11.788726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:11.788076    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:17:11.788726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:11.788140    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:17:12.542317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:12.542279    4291 generic.go:356] "Generic (PLEG): container finished" podID="dfdb1914-f8a5-49c6-bfbb-160bb380737b" containerID="b815e3ccd9c401ae098e77ae1435d5e0039aefd28cdc4f7d647bfbdffffe9077" exitCode=0
Oct 30 13:17:12.542868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:12.542842    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm" event={"ID":"dfdb1914-f8a5-49c6-bfbb-160bb380737b","Type":"ContainerDied","Data":"b815e3ccd9c401ae098e77ae1435d5e0039aefd28cdc4f7d647bfbdffffe9077"}
Oct 30 13:17:12.721629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:12.721597    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:17:12.721836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:12.721664    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:17:12.788555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:12.788519    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Oct 30 13:17:12.788752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:12.788580    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Oct 30 13:17:13.720574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.720532    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:17:13.721075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.720628    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:17:13.793377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.793350    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:13.873789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.872796    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-openshift-ca\") pod \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") "
Oct 30 13:17:13.873789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.872845    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-xks-ca\") pod \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") "
Oct 30 13:17:13.873789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.872968    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dfdb1914-f8a5-49c6-bfbb-160bb380737b-kube-api-access-whnmv\" (UniqueName: \"kubernetes.io/projected/dfdb1914-f8a5-49c6-bfbb-160bb380737b-kube-api-access-whnmv\") pod \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\" (UID: \"dfdb1914-f8a5-49c6-bfbb-160bb380737b\") "
Oct 30 13:17:13.898525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.898176    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-openshift-ca" pod "dfdb1914-f8a5-49c6-bfbb-160bb380737b" (UID: "dfdb1914-f8a5-49c6-bfbb-160bb380737b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:13.902108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.902071    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dfdb1914-f8a5-49c6-bfbb-160bb380737b-kube-api-access-whnmv" pod "dfdb1914-f8a5-49c6-bfbb-160bb380737b" (UID: "dfdb1914-f8a5-49c6-bfbb-160bb380737b"). InnerVolumeSpecName "kube-api-access-whnmv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:13.905424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.905366    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-xks-ca" pod "dfdb1914-f8a5-49c6-bfbb-160bb380737b" (UID: "dfdb1914-f8a5-49c6-bfbb-160bb380737b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:13.981659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.981422    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whnmv\" (UniqueName: \"kubernetes.io/projected/dfdb1914-f8a5-49c6-bfbb-160bb380737b-kube-api-access-whnmv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:13.981659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.981462    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:13.981659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:13.981478    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dfdb1914-f8a5-49c6-bfbb-160bb380737b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:14.369014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.368977    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:14.370741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.370717    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:14.377041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.377009    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 13:17:14.586619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.586579    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"
Oct 30 13:17:14.587287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.587265    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm" event={"ID":"dfdb1914-f8a5-49c6-bfbb-160bb380737b","Type":"ContainerDied","Data":"9f9e71fec2aa0b3e51ac354e7c16c79c9d529af5e47aeed3de7eb017d2062f16"}
Oct 30 13:17:14.587409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.587349    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f9e71fec2aa0b3e51ac354e7c16c79c9d529af5e47aeed3de7eb017d2062f16"
Oct 30 13:17:14.720792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.720704    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:17:14.720792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:14.720761    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:17:15.720653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:15.720608    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:17:15.721210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:15.720692    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:17:16.721306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:16.721268    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:17:16.721780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:16.721328    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:17:17.725954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:17.725925    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:17.726394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:17.726352    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:17.727090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:17.727073    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:17:23.116257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.116175    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"]
Oct 30 13:17:23.116697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.116679    4291 cpu_manager.go:414] "containerMap: removing container" podUID="dfdb1914-f8a5-49c6-bfbb-160bb380737b" containerName="revoke-keys"
Oct 30 13:17:23.116761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.116700    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dfdb1914-f8a5-49c6-bfbb-160bb380737b" containerName="revoke-keys"
Oct 30 13:17:23.116821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.116810    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="dfdb1914-f8a5-49c6-bfbb-160bb380737b" containerName="revoke-keys"
Oct 30 13:17:23.121195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.121178    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.142046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.142004    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"]
Oct 30 13:17:23.149560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.149424    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-52af49-openshift-default-dockercfg-tbzp6\""
Oct 30 13:17:23.212069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212031    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212066    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212190    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212250    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212299    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212359    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212424    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212451    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.212507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.212472    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bscw4\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-kube-api-access-bscw4\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.313965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.313922    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.313965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.313956    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.313989    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314031    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314081    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314120    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314171    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314206    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314243    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bscw4\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-kube-api-access-bscw4\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314500    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314536    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314572    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314603    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.314889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.314870    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.316474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.316455    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.316612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.316594    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.333367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.333330    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.333578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.333552    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bscw4\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-kube-api-access-bscw4\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.455456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.455363    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:23.784443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.784416    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"]
Oct 30 13:17:23.785831 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:23.785800    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90eccdf7_bf8b_45d5_8302_8e7c88eca050.slice/crio-976735cd5546a55388147226bb887a289a1fb04836c64d04d818cd81c665781a WatchSource:0}: Error finding container 976735cd5546a55388147226bb887a289a1fb04836c64d04d818cd81c665781a: Status 404 returned error can't find the container with id 976735cd5546a55388147226bb887a289a1fb04836c64d04d818cd81c665781a
Oct 30 13:17:23.788355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.788323    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:23.788432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.788396    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:23.788469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:23.788431    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:24.584657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.584328    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"]
Oct 30 13:17:24.722617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.722577    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" event={"ID":"90eccdf7-bf8b-45d5-8302-8e7c88eca050","Type":"ContainerStarted","Data":"5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222"}
Oct 30 13:17:24.722927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.722902    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" event={"ID":"90eccdf7-bf8b-45d5-8302-8e7c88eca050","Type":"ContainerStarted","Data":"976735cd5546a55388147226bb887a289a1fb04836c64d04d818cd81c665781a"}
Oct 30 13:17:24.789783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.789729    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podStartSLOduration=1.789713362 podStartE2EDuration="1.789713362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:24.774733425 +0000 UTC m=+855.034971683" watchObservedRunningTime="2026-10-30 13:17:24.789713362 +0000 UTC m=+855.049951630"
Oct 30 13:17:24.791159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.791124    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"]
Oct 30 13:17:24.795472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.795446    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.801743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.801486    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"]
Oct 30 13:17:24.811460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.811428    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:24.811708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.811560    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:17:24.812860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.812832    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:24.812988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.812973    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:24.832650 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.832592    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.832900 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.832878    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhhtb\" (UniqueName: \"kubernetes.io/projected/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-kube-api-access-lhhtb\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.832988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.832950    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.934068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.934033    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.934254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.934088    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.934254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.934207    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhhtb\" (UniqueName: \"kubernetes.io/projected/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-kube-api-access-lhhtb\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.934860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.934836    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.936518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.936497    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:24.979234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:24.979194    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhhtb\" (UniqueName: \"kubernetes.io/projected/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-kube-api-access-lhhtb\") pod \"maas-api-revoke-sub-3915dc57-222hm\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:25.117022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.116986    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:25.442200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.442165    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"]
Oct 30 13:17:25.444909 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:25.444835    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6627a426_e812_4ca7_8bbd_1bd2b5923ba5.slice/crio-6c13fff1b50fd120193579cdda1c6018c278f7f0399ae711c522c89918bb5813 WatchSource:0}: Error finding container 6c13fff1b50fd120193579cdda1c6018c278f7f0399ae711c522c89918bb5813: Status 404 returned error can't find the container with id 6c13fff1b50fd120193579cdda1c6018c278f7f0399ae711c522c89918bb5813
Oct 30 13:17:25.466475 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.466393    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:25.492266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.492219    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:17:25.492559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.492446    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:17:25.730210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.730175    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm" event={"ID":"6627a426-e812-4ca7-8bbd-1bd2b5923ba5","Type":"ContainerStarted","Data":"6c13fff1b50fd120193579cdda1c6018c278f7f0399ae711c522c89918bb5813"}
Oct 30 13:17:25.781696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.781660    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"]
Oct 30 13:17:25.786865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.786840    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.801848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.801815    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"]
Oct 30 13:17:25.844422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.844390    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.844602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.844556    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.844673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.844613    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jgpz\" (UniqueName: \"kubernetes.io/projected/f012eb90-91c8-4430-812a-2be3017b0b57-kube-api-access-6jgpz\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.945890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.945846    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.946092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.945929    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.946092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.946071    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jgpz\" (UniqueName: \"kubernetes.io/projected/f012eb90-91c8-4430-812a-2be3017b0b57-kube-api-access-6jgpz\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.946686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.946619    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.948370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.948339    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:25.979544 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:25.979454    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jgpz\" (UniqueName: \"kubernetes.io/projected/f012eb90-91c8-4430-812a-2be3017b0b57-kube-api-access-6jgpz\") pod \"maas-api-revoke-sub-aa228694-4f2k5\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:26.101604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:26.101558    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:26.436255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:26.436218    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"]
Oct 30 13:17:26.437472 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:26.437445    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf012eb90_91c8_4430_812a_2be3017b0b57.slice/crio-0f94a777d292aa64214bb0bfe838f7b51575bcdc0028b3742f277b1cdc87f7f9 WatchSource:0}: Error finding container 0f94a777d292aa64214bb0bfe838f7b51575bcdc0028b3742f277b1cdc87f7f9: Status 404 returned error can't find the container with id 0f94a777d292aa64214bb0bfe838f7b51575bcdc0028b3742f277b1cdc87f7f9
Oct 30 13:17:26.475192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:26.474790    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:17:26.475192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:26.474844    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:17:26.741795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:26.741758    4291 generic.go:356] "Generic (PLEG): container finished" podID="6627a426-e812-4ca7-8bbd-1bd2b5923ba5" containerID="473bd3eed5b66706d777470131c0ac00db707f1f25d00eef42547bac1e84f27b" exitCode=0
Oct 30 13:17:26.742279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:26.741840    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm" event={"ID":"6627a426-e812-4ca7-8bbd-1bd2b5923ba5","Type":"ContainerDied","Data":"473bd3eed5b66706d777470131c0ac00db707f1f25d00eef42547bac1e84f27b"}
Oct 30 13:17:26.743242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:26.743219    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5" event={"ID":"f012eb90-91c8-4430-812a-2be3017b0b57","Type":"ContainerStarted","Data":"0f94a777d292aa64214bb0bfe838f7b51575bcdc0028b3742f277b1cdc87f7f9"}
Oct 30 13:17:27.460658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:27.460094    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:17:27.460658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:27.460161    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:17:27.778722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:27.778583    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5" event={"ID":"f012eb90-91c8-4430-812a-2be3017b0b57","Type":"ContainerDied","Data":"c753cfafe45b3b40092bd39bc1509284dd18c2d55128213617bf31b11ae88b8c"}
Oct 30 13:17:27.779234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:27.778473    4291 generic.go:356] "Generic (PLEG): container finished" podID="f012eb90-91c8-4430-812a-2be3017b0b57" containerID="c753cfafe45b3b40092bd39bc1509284dd18c2d55128213617bf31b11ae88b8c" exitCode=0
Oct 30 13:17:27.988871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:27.988841    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:28.036964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.036875    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"]
Oct 30 13:17:28.037620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.037599    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6627a426-e812-4ca7-8bbd-1bd2b5923ba5" containerName="revoke-keys"
Oct 30 13:17:28.037707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.037624    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6627a426-e812-4ca7-8bbd-1bd2b5923ba5" containerName="revoke-keys"
Oct 30 13:17:28.037798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.037782    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6627a426-e812-4ca7-8bbd-1bd2b5923ba5" containerName="revoke-keys"
Oct 30 13:17:28.041171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.041149    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.070606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.062464    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"]
Oct 30 13:17:28.070606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.062860    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-f9ebb5-openshift-default-dockercfg-259p4\""
Oct 30 13:17:28.093598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.093560    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-kube-api-access-lhhtb\" (UniqueName: \"kubernetes.io/projected/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-kube-api-access-lhhtb\") pod \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") "
Oct 30 13:17:28.093833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.093808    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-openshift-ca\") pod \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") "
Oct 30 13:17:28.093921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.093866    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-xks-ca\") pod \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\" (UID: \"6627a426-e812-4ca7-8bbd-1bd2b5923ba5\") "
Oct 30 13:17:28.095964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.095924    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-openshift-ca" pod "6627a426-e812-4ca7-8bbd-1bd2b5923ba5" (UID: "6627a426-e812-4ca7-8bbd-1bd2b5923ba5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:28.096988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.096959    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-xks-ca" pod "6627a426-e812-4ca7-8bbd-1bd2b5923ba5" (UID: "6627a426-e812-4ca7-8bbd-1bd2b5923ba5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:28.097383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.097359    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-kube-api-access-lhhtb" pod "6627a426-e812-4ca7-8bbd-1bd2b5923ba5" (UID: "6627a426-e812-4ca7-8bbd-1bd2b5923ba5"). InnerVolumeSpecName "kube-api-access-lhhtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:28.195389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195350    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195405    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srzmg\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-kube-api-access-srzmg\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195491    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195602    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195655    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195695    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195768    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195793    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195813    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.195965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195931    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:28.195965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195959    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:28.196059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.195976    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhhtb\" (UniqueName: \"kubernetes.io/projected/6627a426-e812-4ca7-8bbd-1bd2b5923ba5-kube-api-access-lhhtb\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:28.297362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297258    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srzmg\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-kube-api-access-srzmg\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297317    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297362    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297427    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297486    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297547    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297589    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297625    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297742    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297930    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.297980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.297941    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.298173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.298151    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.298248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.298181    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.298617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.298593    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.300071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.300043    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.300205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.300190    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.317556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.317519    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.317705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.317532    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srzmg\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-kube-api-access-srzmg\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.391232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.391174    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:28.459657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.459054    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:17:28.459657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.459114    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:17:28.741804 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:28.741756    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bb2aeb5_62a0_439d_818d_1bb8c9507942.slice/crio-1cbeadd6f7e8cc8cfa31ba3fbbf925a8729e943b4184fcd2a5bc4bd1e6833f65 WatchSource:0}: Error finding container 1cbeadd6f7e8cc8cfa31ba3fbbf925a8729e943b4184fcd2a5bc4bd1e6833f65: Status 404 returned error can't find the container with id 1cbeadd6f7e8cc8cfa31ba3fbbf925a8729e943b4184fcd2a5bc4bd1e6833f65
Oct 30 13:17:28.741946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.741834    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"]
Oct 30 13:17:28.744378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.744343    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:28.744511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.744410    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:28.744511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.744444    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:28.797997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.797964    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" event={"ID":"8bb2aeb5-62a0-439d-818d-1bb8c9507942","Type":"ContainerStarted","Data":"1cbeadd6f7e8cc8cfa31ba3fbbf925a8729e943b4184fcd2a5bc4bd1e6833f65"}
Oct 30 13:17:28.800073 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.800043    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"
Oct 30 13:17:28.800238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.800058    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm" event={"ID":"6627a426-e812-4ca7-8bbd-1bd2b5923ba5","Type":"ContainerDied","Data":"6c13fff1b50fd120193579cdda1c6018c278f7f0399ae711c522c89918bb5813"}
Oct 30 13:17:28.800354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:28.800341    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c13fff1b50fd120193579cdda1c6018c278f7f0399ae711c522c89918bb5813"
Oct 30 13:17:29.092336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.092298    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:29.214032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.213807    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-openshift-ca\") pod \"f012eb90-91c8-4430-812a-2be3017b0b57\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") "
Oct 30 13:17:29.214372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.214354    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f012eb90-91c8-4430-812a-2be3017b0b57-kube-api-access-6jgpz\" (UniqueName: \"kubernetes.io/projected/f012eb90-91c8-4430-812a-2be3017b0b57-kube-api-access-6jgpz\") pod \"f012eb90-91c8-4430-812a-2be3017b0b57\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") "
Oct 30 13:17:29.214472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.214462    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-xks-ca\") pod \"f012eb90-91c8-4430-812a-2be3017b0b57\" (UID: \"f012eb90-91c8-4430-812a-2be3017b0b57\") "
Oct 30 13:17:29.215436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.215417    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-openshift-ca" pod "f012eb90-91c8-4430-812a-2be3017b0b57" (UID: "f012eb90-91c8-4430-812a-2be3017b0b57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:29.216841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.216817    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-xks-ca" pod "f012eb90-91c8-4430-812a-2be3017b0b57" (UID: "f012eb90-91c8-4430-812a-2be3017b0b57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:29.217830 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.217804    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f012eb90-91c8-4430-812a-2be3017b0b57-kube-api-access-6jgpz" pod "f012eb90-91c8-4430-812a-2be3017b0b57" (UID: "f012eb90-91c8-4430-812a-2be3017b0b57"). InnerVolumeSpecName "kube-api-access-6jgpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:29.317737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.316680    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:29.317737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.316719    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jgpz\" (UniqueName: \"kubernetes.io/projected/f012eb90-91c8-4430-812a-2be3017b0b57-kube-api-access-6jgpz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:29.317737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.316735    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f012eb90-91c8-4430-812a-2be3017b0b57-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:29.470660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.470064    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:17:29.470660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.470126    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:17:29.814339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.814303    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" event={"ID":"8bb2aeb5-62a0-439d-818d-1bb8c9507942","Type":"ContainerStarted","Data":"118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f"}
Oct 30 13:17:29.815782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.815762    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"
Oct 30 13:17:29.815875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.815811    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5" event={"ID":"f012eb90-91c8-4430-812a-2be3017b0b57","Type":"ContainerDied","Data":"0f94a777d292aa64214bb0bfe838f7b51575bcdc0028b3742f277b1cdc87f7f9"}
Oct 30 13:17:29.815875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.815856    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f94a777d292aa64214bb0bfe838f7b51575bcdc0028b3742f277b1cdc87f7f9"
Oct 30 13:17:29.912573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:29.909402    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podStartSLOduration=1.909376994 podStartE2EDuration="1.909376994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:29.871127808 +0000 UTC m=+860.131366071" watchObservedRunningTime="2026-10-30 13:17:29.909376994 +0000 UTC m=+860.169615229"
Oct 30 13:17:30.391797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:30.391754    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:30.393328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:30.393307    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:17:30.393403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:30.393353    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 13:17:30.460661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:30.458085    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Oct 30 13:17:30.460661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:30.458154    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Oct 30 13:17:31.391604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.391560    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:17:31.392142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.391628    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 13:17:31.608440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.608409    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:31.608849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.608826    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:31.609594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.609573    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:17:31.698936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.698847    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"]
Oct 30 13:17:31.699338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.699324    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f012eb90-91c8-4430-812a-2be3017b0b57" containerName="revoke-keys"
Oct 30 13:17:31.699376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.699339    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f012eb90-91c8-4430-812a-2be3017b0b57" containerName="revoke-keys"
Oct 30 13:17:31.699439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.699429    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f012eb90-91c8-4430-812a-2be3017b0b57" containerName="revoke-keys"
Oct 30 13:17:31.702763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.702744    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.718556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.718523    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:31.719550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.718833    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:31.719550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.718969    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:31.719550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.719002    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:17:31.721659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.721577    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"]
Oct 30 13:17:31.872661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.870935    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.872661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.872145    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rm4s9\" (UniqueName: \"kubernetes.io/projected/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-kube-api-access-rm4s9\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.872661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.872381    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.977654 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.977532    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.977934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.977915    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rm4s9\" (UniqueName: \"kubernetes.io/projected/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-kube-api-access-rm4s9\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.978083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.978069    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.978305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.978276    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.981282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.981250    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:31.997944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:31.997907    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rm4s9\" (UniqueName: \"kubernetes.io/projected/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-kube-api-access-rm4s9\") pod \"maas-api-revoke-sub-e20ae2c5-2dgbv\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:32.016585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:32.016554    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:32.347859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:32.347828    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"]
Oct 30 13:17:32.349479 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:32.349448    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb75cfaf0_e6a2_4d2c_b21e_c2a80ac8ab46.slice/crio-44d22a19cb98fb43873269bc57fc8260887294683e8c41d7ecac4da2feaf3bc5 WatchSource:0}: Error finding container 44d22a19cb98fb43873269bc57fc8260887294683e8c41d7ecac4da2feaf3bc5: Status 404 returned error can't find the container with id 44d22a19cb98fb43873269bc57fc8260887294683e8c41d7ecac4da2feaf3bc5
Oct 30 13:17:32.391516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:32.391491    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:17:32.391591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:32.391533    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 13:17:32.837990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:32.837958    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv" event={"ID":"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46","Type":"ContainerStarted","Data":"b636d05e7edef43abc2d23151ba5c418fc94ba52960eaf3eeb071f6a12d63dae"}
Oct 30 13:17:32.838387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:32.838016    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv" event={"ID":"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46","Type":"ContainerStarted","Data":"44d22a19cb98fb43873269bc57fc8260887294683e8c41d7ecac4da2feaf3bc5"}
Oct 30 13:17:32.899816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:32.899760    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv" podStartSLOduration=1.89974507 podStartE2EDuration="1.89974507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:32.895692058 +0000 UTC m=+863.155930292" watchObservedRunningTime="2026-10-30 13:17:32.89974507 +0000 UTC m=+863.159983304"
Oct 30 13:17:33.392095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:33.392062    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:17:33.392266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:33.392126    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 13:17:33.846003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:33.845971    4291 generic.go:356] "Generic (PLEG): container finished" podID="b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" containerID="b636d05e7edef43abc2d23151ba5c418fc94ba52960eaf3eeb071f6a12d63dae" exitCode=0
Oct 30 13:17:33.846383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:33.846043    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv" event={"ID":"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46","Type":"ContainerDied","Data":"b636d05e7edef43abc2d23151ba5c418fc94ba52960eaf3eeb071f6a12d63dae"}
Oct 30 13:17:34.391845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:34.391808    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:17:34.392021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:34.391887    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 13:17:35.049057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.049029    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:35.214439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.214347    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-xks-ca\") pod \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") "
Oct 30 13:17:35.214619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.214600    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-kube-api-access-rm4s9\" (UniqueName: \"kubernetes.io/projected/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-kube-api-access-rm4s9\") pod \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") "
Oct 30 13:17:35.214692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.214667    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-openshift-ca\") pod \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\" (UID: \"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46\") "
Oct 30 13:17:35.215106 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.215072    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-openshift-ca" pod "b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" (UID: "b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:35.216504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.216473    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-xks-ca" pod "b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" (UID: "b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:35.216668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.216629    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-kube-api-access-rm4s9" pod "b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" (UID: "b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46"). InnerVolumeSpecName "kube-api-access-rm4s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:35.316244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.316206    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.316244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.316240    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rm4s9\" (UniqueName: \"kubernetes.io/projected/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-kube-api-access-rm4s9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.316244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.316252    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:35.391968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.391931    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused" start-of-body=
Oct 30 13:17:35.392168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.392004    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.74:15021/healthz/ready\": dial tcp 10.133.0.74:15021: connect: connection refused"
Oct 30 13:17:35.884754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.884716    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv" event={"ID":"b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46","Type":"ContainerDied","Data":"44d22a19cb98fb43873269bc57fc8260887294683e8c41d7ecac4da2feaf3bc5"}
Oct 30 13:17:35.884754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.884747    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"
Oct 30 13:17:35.885007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:35.884774    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44d22a19cb98fb43873269bc57fc8260887294683e8c41d7ecac4da2feaf3bc5"
Oct 30 13:17:36.126916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.126874    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"]
Oct 30 13:17:36.127471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.127450    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" containerName="revoke-keys"
Oct 30 13:17:36.127541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.127472    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" containerName="revoke-keys"
Oct 30 13:17:36.127626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.127615    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" containerName="revoke-keys"
Oct 30 13:17:36.130924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.130903    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.150798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.148462    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"]
Oct 30 13:17:36.150798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.149170    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:36.150798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.149275    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:17:36.155413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.155369    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:36.155413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.155381    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:36.228731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.228689    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.228942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.228860    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.228942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.228888    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdp9r\" (UniqueName: \"kubernetes.io/projected/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-kube-api-access-vdp9r\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.330168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.330135    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.330324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.330248    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.330324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.330278    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdp9r\" (UniqueName: \"kubernetes.io/projected/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-kube-api-access-vdp9r\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.330939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.330914    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.333002 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.332971    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.349753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.349720    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdp9r\" (UniqueName: \"kubernetes.io/projected/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-kube-api-access-vdp9r\") pod \"maas-api-revoke-sub-2ce0e250-qd86g\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.446363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.446268    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:36.446711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.446686    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:36.447526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.447505    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:17:36.453767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.453734    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:36.759405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.759374    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"]
Oct 30 13:17:36.760006 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:36.759974    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0472bdc0_8c4c_4970_9800_dd95c5b9feb6.slice/crio-12e8683a743c5215b74bc6b29a7b402e1595a402e311c621739d20e7b06d3729 WatchSource:0}: Error finding container 12e8683a743c5215b74bc6b29a7b402e1595a402e311c621739d20e7b06d3729: Status 404 returned error can't find the container with id 12e8683a743c5215b74bc6b29a7b402e1595a402e311c621739d20e7b06d3729
Oct 30 13:17:36.892498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:36.892463    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g" event={"ID":"0472bdc0-8c4c-4970-9800-dd95c5b9feb6","Type":"ContainerStarted","Data":"12e8683a743c5215b74bc6b29a7b402e1595a402e311c621739d20e7b06d3729"}
Oct 30 13:17:37.100855 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:37.100821    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:17:37.901212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:37.901107    4291 generic.go:356] "Generic (PLEG): container finished" podID="0472bdc0-8c4c-4970-9800-dd95c5b9feb6" containerID="dd7db99aa213167bc8b5786e228eb4bd14d81d7f217a8c2d835ff63c25fa4078" exitCode=0
Oct 30 13:17:37.901212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:37.901193    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g" event={"ID":"0472bdc0-8c4c-4970-9800-dd95c5b9feb6","Type":"ContainerDied","Data":"dd7db99aa213167bc8b5786e228eb4bd14d81d7f217a8c2d835ff63c25fa4078"}
Oct 30 13:17:38.194866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.194783    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"]
Oct 30 13:17:38.199281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.199254    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.222692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.219232    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"]
Oct 30 13:17:38.232543 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:17:38.231826    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:17:38.357662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357431    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.357662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357495    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.357662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357541    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.357662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357620    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.357662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357668    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8157e3-282b-444e-ad9e-52c201f99492-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.358063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357697    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.358063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357762    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8157e3-282b-444e-ad9e-52c201f99492-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.358063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357830    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.358063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.357892    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgsmj\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-kube-api-access-cgsmj\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.459725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459360    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8157e3-282b-444e-ad9e-52c201f99492-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.459725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459449    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.459725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459502    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgsmj\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-kube-api-access-cgsmj\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.459725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459614    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.459725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459661    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.459725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459706    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.460207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459782    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.460207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459813    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8157e3-282b-444e-ad9e-52c201f99492-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.460207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.459846    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.461672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.460733    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.461672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.461397    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8157e3-282b-444e-ad9e-52c201f99492-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.465661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.462011    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.465661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.462275    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.465661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.465444    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.485670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.483266    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.495678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.493363    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8157e3-282b-444e-ad9e-52c201f99492-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.509713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.508098    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgsmj\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-kube-api-access-cgsmj\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.509713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.508890    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:38.528785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:38.524874    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:39.493292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:39.493252    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:39.493888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:39.493863    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:39.494063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:39.494031    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:39.525675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:39.519474    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"]
Oct 30 13:17:39.633682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:39.632841    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x" podUID="96115c1b-7ade-4f59-ad95-afdf3ba11a01" containerName="main" containerID="cri-o://351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de" gracePeriod=45
Oct 30 13:17:39.735985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:39.729339    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x" podUID="96115c1b-7ade-4f59-ad95-afdf3ba11a01" containerName="main" probeResult="failure" output="Get \"https://10.133.0.56:8000/ready\": dial tcp 10.133.0.56:8000: connect: connection refused"
Oct 30 13:17:40.195334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.194608    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g" event={"ID":"0472bdc0-8c4c-4970-9800-dd95c5b9feb6","Type":"ContainerDied","Data":"12e8683a743c5215b74bc6b29a7b402e1595a402e311c621739d20e7b06d3729"}
Oct 30 13:17:40.195334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.194693    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12e8683a743c5215b74bc6b29a7b402e1595a402e311c621739d20e7b06d3729"
Oct 30 13:17:40.197414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.197102    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:40.215469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.215239    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" event={"ID":"1a8157e3-282b-444e-ad9e-52c201f99492","Type":"ContainerStarted","Data":"2ed6f2bdff2d8dac481244a4cf8f4a82990675c4654624b86657333a8d6039f6"}
Oct 30 13:17:40.381140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.356780    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-xks-ca\") pod \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") "
Oct 30 13:17:40.381140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.356973    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-kube-api-access-vdp9r\" (UniqueName: \"kubernetes.io/projected/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-kube-api-access-vdp9r\") pod \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") "
Oct 30 13:17:40.381140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.364063    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-openshift-ca\") pod \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\" (UID: \"0472bdc0-8c4c-4970-9800-dd95c5b9feb6\") "
Oct 30 13:17:40.381140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.380824    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-openshift-ca" pod "0472bdc0-8c4c-4970-9800-dd95c5b9feb6" (UID: "0472bdc0-8c4c-4970-9800-dd95c5b9feb6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:40.436685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.428887    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-kube-api-access-vdp9r" pod "0472bdc0-8c4c-4970-9800-dd95c5b9feb6" (UID: "0472bdc0-8c4c-4970-9800-dd95c5b9feb6"). InnerVolumeSpecName "kube-api-access-vdp9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:40.436685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.430422    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-xks-ca" pod "0472bdc0-8c4c-4970-9800-dd95c5b9feb6" (UID: "0472bdc0-8c4c-4970-9800-dd95c5b9feb6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:40.469658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.466672    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.469658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.466716    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdp9r\" (UniqueName: \"kubernetes.io/projected/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-kube-api-access-vdp9r\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.469658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.466733    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0472bdc0-8c4c-4970-9800-dd95c5b9feb6-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.610216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.610144    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:17:40.769765 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.769724    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-home\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-home\") pod \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") "
Oct 30 13:17:40.769917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.769810    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-model-cache\") pod \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") "
Oct 30 13:17:40.769954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.769923    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tmp-dir\") pod \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") "
Oct 30 13:17:40.769986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.769965    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96115c1b-7ade-4f59-ad95-afdf3ba11a01-kube-api-access-8pm6s\" (UniqueName: \"kubernetes.io/projected/96115c1b-7ade-4f59-ad95-afdf3ba11a01-kube-api-access-8pm6s\") pod \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") "
Oct 30 13:17:40.770062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770000    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tls-certs\" (UniqueName: \"kubernetes.io/secret/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tls-certs\") pod \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") "
Oct 30 13:17:40.770062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770055    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-dshm\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-dshm\") pod \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\" (UID: \"96115c1b-7ade-4f59-ad95-afdf3ba11a01\") "
Oct 30 13:17:40.770168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770056    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-home" pod "96115c1b-7ade-4f59-ad95-afdf3ba11a01" (UID: "96115c1b-7ade-4f59-ad95-afdf3ba11a01"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:40.770168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770065    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-model-cache" pod "96115c1b-7ade-4f59-ad95-afdf3ba11a01" (UID: "96115c1b-7ade-4f59-ad95-afdf3ba11a01"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:40.770263 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770185    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tmp-dir" pod "96115c1b-7ade-4f59-ad95-afdf3ba11a01" (UID: "96115c1b-7ade-4f59-ad95-afdf3ba11a01"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:40.770492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770470    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.770594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770493    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.770594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.770509    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.772369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.772335    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tls-certs" pod "96115c1b-7ade-4f59-ad95-afdf3ba11a01" (UID: "96115c1b-7ade-4f59-ad95-afdf3ba11a01"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:40.772483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.772404    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-dshm" pod "96115c1b-7ade-4f59-ad95-afdf3ba11a01" (UID: "96115c1b-7ade-4f59-ad95-afdf3ba11a01"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:17:40.772483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.772434    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96115c1b-7ade-4f59-ad95-afdf3ba11a01-kube-api-access-8pm6s" pod "96115c1b-7ade-4f59-ad95-afdf3ba11a01" (UID: "96115c1b-7ade-4f59-ad95-afdf3ba11a01"). InnerVolumeSpecName "kube-api-access-8pm6s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:40.871485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.871392    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pm6s\" (UniqueName: \"kubernetes.io/projected/96115c1b-7ade-4f59-ad95-afdf3ba11a01-kube-api-access-8pm6s\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.871485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.871436    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/96115c1b-7ade-4f59-ad95-afdf3ba11a01-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:40.871485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:40.871452    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/96115c1b-7ade-4f59-ad95-afdf3ba11a01-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:41.224265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.224175    4291 generic.go:356] "Generic (PLEG): container finished" podID="96115c1b-7ade-4f59-ad95-afdf3ba11a01" containerID="351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de" exitCode=0
Oct 30 13:17:41.224265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.224221    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"
Oct 30 13:17:41.224488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.224263    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x" event={"ID":"96115c1b-7ade-4f59-ad95-afdf3ba11a01","Type":"ContainerDied","Data":"351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de"}
Oct 30 13:17:41.224488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.224304    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x" event={"ID":"96115c1b-7ade-4f59-ad95-afdf3ba11a01","Type":"ContainerDied","Data":"ac42197a9592a90ad6a8838ad629107db6cdf8dd548b59e8be0636212126d052"}
Oct 30 13:17:41.224488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.224325    4291 scope.go:122] "RemoveContainer" containerID="351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de"
Oct 30 13:17:41.226401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.226370    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" event={"ID":"1a8157e3-282b-444e-ad9e-52c201f99492","Type":"ContainerStarted","Data":"c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c"}
Oct 30 13:17:41.226557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.226539    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"
Oct 30 13:17:41.236268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.236050    4291 scope.go:122] "RemoveContainer" containerID="351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de"
Oct 30 13:17:41.236540 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:17:41.236518    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de\": container with ID starting with 351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de not found: ID does not exist" containerID="351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de"
Oct 30 13:17:41.236621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.236551    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de"} err="failed to get container status \"351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de\": rpc error: code = NotFound desc = could not find container \"351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de\": container with ID starting with 351f4c8c1775e6437ef664def990c443962c2abfc7cb3fd30f5668b53ad895de not found: ID does not exist"
Oct 30 13:17:41.305565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.305511    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podStartSLOduration=3.305493632 podStartE2EDuration="3.305493632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:41.303596972 +0000 UTC m=+871.563835225" watchObservedRunningTime="2026-10-30 13:17:41.305493632 +0000 UTC m=+871.565731865"
Oct 30 13:17:41.347768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.347712    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"]
Oct 30 13:17:41.366661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.350074    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-9bf8ad/e2e-conflict-a-dfdb6918-kserve-6d8dbc45dc-gsf4x"]
Oct 30 13:17:41.525560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.525471    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:41.527070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.527041    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:17:41.527186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.527089    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:17:41.881847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:41.881808    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96115c1b-7ade-4f59-ad95-afdf3ba11a01" path="/var/lib/kubelet/pods/96115c1b-7ade-4f59-ad95-afdf3ba11a01/volumes"
Oct 30 13:17:42.526107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:42.526069    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:17:42.526301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:42.526126    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:17:43.108140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.108105    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"]
Oct 30 13:17:43.108562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.108549    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0472bdc0-8c4c-4970-9800-dd95c5b9feb6" containerName="revoke-keys"
Oct 30 13:17:43.108600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.108563    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0472bdc0-8c4c-4970-9800-dd95c5b9feb6" containerName="revoke-keys"
Oct 30 13:17:43.108600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.108577    4291 cpu_manager.go:414] "containerMap: removing container" podUID="96115c1b-7ade-4f59-ad95-afdf3ba11a01" containerName="main"
Oct 30 13:17:43.108600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.108582    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96115c1b-7ade-4f59-ad95-afdf3ba11a01" containerName="main"
Oct 30 13:17:43.108755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.108687    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0472bdc0-8c4c-4970-9800-dd95c5b9feb6" containerName="revoke-keys"
Oct 30 13:17:43.108755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.108697    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="96115c1b-7ade-4f59-ad95-afdf3ba11a01" containerName="main"
Oct 30 13:17:43.113486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.113466    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.134732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.134696    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"]
Oct 30 13:17:43.201572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201529    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.201749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201661    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9t4zm\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-kube-api-access-9t4zm\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.201749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201703    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.201749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201727    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.201891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201752    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.201891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201855    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.201993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201893    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.202044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.201995    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.202092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.202054    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.302825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.302791    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.302861    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.302906    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9t4zm\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-kube-api-access-9t4zm\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.302929    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.302961    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.302994    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303042    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303082    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303142    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303360    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303562    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303572    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303725    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.303815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.303732    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.305456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.305433    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.305894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.305877    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.325197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.325160    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.325330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.325237    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9t4zm\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-kube-api-access-9t4zm\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.430243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.430150    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:43.526676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.526327    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:17:43.526676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.526388    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:17:43.785188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.785160    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"]
Oct 30 13:17:43.785724 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:43.785698    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac358606_1b97_4eb2_b53a_6b731b9aa36f.slice/crio-68bae72cbff2fcd15d8895e0ba3e4f38454812f53c3c75cd301e6a8877334e4f WatchSource:0}: Error finding container 68bae72cbff2fcd15d8895e0ba3e4f38454812f53c3c75cd301e6a8877334e4f: Status 404 returned error can't find the container with id 68bae72cbff2fcd15d8895e0ba3e4f38454812f53c3c75cd301e6a8877334e4f
Oct 30 13:17:43.788268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.788235    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:43.788344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.788300    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:43.788344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:43.788329    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:44.253659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.249676    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" event={"ID":"ac358606-1b97-4eb2-b53a-6b731b9aa36f","Type":"ContainerStarted","Data":"89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c"}
Oct 30 13:17:44.253659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.249750    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" event={"ID":"ac358606-1b97-4eb2-b53a-6b731b9aa36f","Type":"ContainerStarted","Data":"68bae72cbff2fcd15d8895e0ba3e4f38454812f53c3c75cd301e6a8877334e4f"}
Oct 30 13:17:44.299992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.299925    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podStartSLOduration=1.29990579 podStartE2EDuration="1.29990579s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:44.297445268 +0000 UTC m=+874.557683519" watchObservedRunningTime="2026-10-30 13:17:44.29990579 +0000 UTC m=+874.560144025"
Oct 30 13:17:44.431194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.431160    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:44.432466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.432443    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:17:44.432667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.432622    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 13:17:44.526500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.526420    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:17:44.526500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:44.526474    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:17:45.430908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:45.430868    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:17:45.431487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:45.431457    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 13:17:45.525941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:45.525914    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:17:45.526126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:45.526108    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:17:46.094661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.090775    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"]
Oct 30 13:17:46.099667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.098093    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.117153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.116715    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"]
Oct 30 13:17:46.117153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.116917    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:17:46.117153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.116993    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:46.117153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.117119    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:46.125882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.125857    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:46.240243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.240207    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.240425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.240329    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmbsv\" (UniqueName: \"kubernetes.io/projected/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-kube-api-access-cmbsv\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.240425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.240398    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.341048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.341005    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.341243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.341130    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.341690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.341667    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmbsv\" (UniqueName: \"kubernetes.io/projected/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-kube-api-access-cmbsv\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.341846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.341669    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.343432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.343410    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.369720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.366338    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmbsv\" (UniqueName: \"kubernetes.io/projected/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-kube-api-access-cmbsv\") pod \"maas-api-revoke-sub-eac47664-72zjf\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.412662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.409273    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"]
Oct 30 13:17:46.414869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.414218    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.428268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.427530    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"]
Oct 30 13:17:46.428268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.427755    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:46.432712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.431307    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:17:46.432712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.431346    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 13:17:46.530690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.525734    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:17:46.530690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.525800    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:17:46.546127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.545179    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.546127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.545230    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.546127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.545256    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt7qr\" (UniqueName: \"kubernetes.io/projected/625617c8-abd6-4512-8da7-13847dc2df39-kube-api-access-tt7qr\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.647064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.646981    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.647064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.647054    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.647317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.647104    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt7qr\" (UniqueName: \"kubernetes.io/projected/625617c8-abd6-4512-8da7-13847dc2df39-kube-api-access-tt7qr\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.648056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.648026    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.653665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.650170    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.681666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.668597    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt7qr\" (UniqueName: \"kubernetes.io/projected/625617c8-abd6-4512-8da7-13847dc2df39-kube-api-access-tt7qr\") pod \"maas-api-revoke-sub-e699e426-r4nlf\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.735409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.735118    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:46.886981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:46.886957    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"]
Oct 30 13:17:47.076662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:47.072266    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"]
Oct 30 13:17:47.324656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:47.322324    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf" event={"ID":"2c2e2347-a7af-46e0-a450-94b9bdf15ed5","Type":"ContainerStarted","Data":"ce51a390f2f9eec073b7f9a37e019647f74f73d03f20d6d7b8377f3e23f0d173"}
Oct 30 13:17:47.324656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:47.324050    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf" event={"ID":"625617c8-abd6-4512-8da7-13847dc2df39","Type":"ContainerStarted","Data":"a946a10f6886944d54c9131fc596ca3c721f9dd81cd4009020a9961242d5683c"}
Oct 30 13:17:47.431603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:47.431562    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:17:47.431842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:47.431624    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 13:17:47.526618 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:47.526582    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Oct 30 13:17:47.527045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:47.526651    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Oct 30 13:17:48.333007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.332960    4291 generic.go:356] "Generic (PLEG): container finished" podID="625617c8-abd6-4512-8da7-13847dc2df39" containerID="2325c4727d27ff55030ddf385300cb291b22b389335d3a1758d63347933f249f" exitCode=0
Oct 30 13:17:48.333208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.333044    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf" event={"ID":"625617c8-abd6-4512-8da7-13847dc2df39","Type":"ContainerDied","Data":"2325c4727d27ff55030ddf385300cb291b22b389335d3a1758d63347933f249f"}
Oct 30 13:17:48.335211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.335177    4291 generic.go:356] "Generic (PLEG): container finished" podID="2c2e2347-a7af-46e0-a450-94b9bdf15ed5" containerID="fedac7dc1fd3cd41e70b01bec32af5034eac9c78f0e95c925bad368af7c892b4" exitCode=0
Oct 30 13:17:48.335319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.335246    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf" event={"ID":"2c2e2347-a7af-46e0-a450-94b9bdf15ed5","Type":"ContainerDied","Data":"fedac7dc1fd3cd41e70b01bec32af5034eac9c78f0e95c925bad368af7c892b4"}
Oct 30 13:17:48.430982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.430893    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:17:48.430982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.430957    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 13:17:48.530187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.530151    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:48.530680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.530555    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:48.531388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:48.531366    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:17:49.432583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.431950    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:17:49.432583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.432012    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 13:17:49.587240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.586910    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:49.613250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.613227    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:49.717448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.717402    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-openshift-ca\") pod \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") "
Oct 30 13:17:49.717676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.717486    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-xks-ca\") pod \"625617c8-abd6-4512-8da7-13847dc2df39\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") "
Oct 30 13:17:49.717676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.717545    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-kube-api-access-cmbsv\" (UniqueName: \"kubernetes.io/projected/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-kube-api-access-cmbsv\") pod \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") "
Oct 30 13:17:49.717676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.717629    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/625617c8-abd6-4512-8da7-13847dc2df39-kube-api-access-tt7qr\" (UniqueName: \"kubernetes.io/projected/625617c8-abd6-4512-8da7-13847dc2df39-kube-api-access-tt7qr\") pod \"625617c8-abd6-4512-8da7-13847dc2df39\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") "
Oct 30 13:17:49.717866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.717717    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-openshift-ca\") pod \"625617c8-abd6-4512-8da7-13847dc2df39\" (UID: \"625617c8-abd6-4512-8da7-13847dc2df39\") "
Oct 30 13:17:49.717866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.717767    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-xks-ca\") pod \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\" (UID: \"2c2e2347-a7af-46e0-a450-94b9bdf15ed5\") "
Oct 30 13:17:49.717958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.717881    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-openshift-ca" pod "2c2e2347-a7af-46e0-a450-94b9bdf15ed5" (UID: "2c2e2347-a7af-46e0-a450-94b9bdf15ed5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:49.718163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.718137    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-openshift-ca" pod "625617c8-abd6-4512-8da7-13847dc2df39" (UID: "625617c8-abd6-4512-8da7-13847dc2df39"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:49.718387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.718367    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:49.718460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.718393    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:49.719995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.719940    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-xks-ca" pod "625617c8-abd6-4512-8da7-13847dc2df39" (UID: "625617c8-abd6-4512-8da7-13847dc2df39"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:49.719995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.719989    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-xks-ca" pod "2c2e2347-a7af-46e0-a450-94b9bdf15ed5" (UID: "2c2e2347-a7af-46e0-a450-94b9bdf15ed5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:49.720129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.719989    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-kube-api-access-cmbsv" pod "2c2e2347-a7af-46e0-a450-94b9bdf15ed5" (UID: "2c2e2347-a7af-46e0-a450-94b9bdf15ed5"). InnerVolumeSpecName "kube-api-access-cmbsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:49.720129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.720063    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/625617c8-abd6-4512-8da7-13847dc2df39-kube-api-access-tt7qr" pod "625617c8-abd6-4512-8da7-13847dc2df39" (UID: "625617c8-abd6-4512-8da7-13847dc2df39"). InnerVolumeSpecName "kube-api-access-tt7qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:49.819148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.819114    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:49.819148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.819144    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/625617c8-abd6-4512-8da7-13847dc2df39-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:49.819148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.819153    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmbsv\" (UniqueName: \"kubernetes.io/projected/2c2e2347-a7af-46e0-a450-94b9bdf15ed5-kube-api-access-cmbsv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:49.819463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:49.819163    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt7qr\" (UniqueName: \"kubernetes.io/projected/625617c8-abd6-4512-8da7-13847dc2df39-kube-api-access-tt7qr\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:50.348644 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.348613    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"
Oct 30 13:17:50.348852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.348664    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf" event={"ID":"2c2e2347-a7af-46e0-a450-94b9bdf15ed5","Type":"ContainerDied","Data":"ce51a390f2f9eec073b7f9a37e019647f74f73d03f20d6d7b8377f3e23f0d173"}
Oct 30 13:17:50.348852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.348728    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce51a390f2f9eec073b7f9a37e019647f74f73d03f20d6d7b8377f3e23f0d173"
Oct 30 13:17:50.350348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.350319    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"
Oct 30 13:17:50.350480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.350375    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf" event={"ID":"625617c8-abd6-4512-8da7-13847dc2df39","Type":"ContainerDied","Data":"a946a10f6886944d54c9131fc596ca3c721f9dd81cd4009020a9961242d5683c"}
Oct 30 13:17:50.350480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.350424    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a946a10f6886944d54c9131fc596ca3c721f9dd81cd4009020a9961242d5683c"
Oct 30 13:17:50.430734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.430695    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused" start-of-body=
Oct 30 13:17:50.430946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.430748    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.78:15021/healthz/ready\": dial tcp 10.133.0.78:15021: connect: connection refused"
Oct 30 13:17:50.526981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.526946    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"]
Oct 30 13:17:50.527424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.527410    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2c2e2347-a7af-46e0-a450-94b9bdf15ed5" containerName="revoke-keys"
Oct 30 13:17:50.527474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.527429    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c2e2347-a7af-46e0-a450-94b9bdf15ed5" containerName="revoke-keys"
Oct 30 13:17:50.527474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.527451    4291 cpu_manager.go:414] "containerMap: removing container" podUID="625617c8-abd6-4512-8da7-13847dc2df39" containerName="revoke-keys"
Oct 30 13:17:50.527474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.527459    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="625617c8-abd6-4512-8da7-13847dc2df39" containerName="revoke-keys"
Oct 30 13:17:50.527583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.527573    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c2e2347-a7af-46e0-a450-94b9bdf15ed5" containerName="revoke-keys"
Oct 30 13:17:50.527617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.527584    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="625617c8-abd6-4512-8da7-13847dc2df39" containerName="revoke-keys"
Oct 30 13:17:50.531013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.530991    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.549987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.549956    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:50.550133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.550047    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:50.550133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.550102    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:50.552318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.552289    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"]
Oct 30 13:17:50.558698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.558672    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:17:50.627353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.627257    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.627353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.627350    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v6vm\" (UniqueName: \"kubernetes.io/projected/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-kube-api-access-7v6vm\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.628151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.627369    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.728150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.728111    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.728328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.728228    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v6vm\" (UniqueName: \"kubernetes.io/projected/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-kube-api-access-7v6vm\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.728328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.728249    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.728947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.728911    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.731189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.731161    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.747939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.747905    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v6vm\" (UniqueName: \"kubernetes.io/projected/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-kube-api-access-7v6vm\") pod \"maas-api-revoke-sub-bf80ac2c-d8kpf\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:50.859552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:50.859515    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:51.178438 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:51.178350    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a45093_dc1e_41f9_b7b1_b3d009c75d8c.slice/crio-57df00648526679e84aa214e291f0907b435ac1b35862ec837ed24b3d265dfe8 WatchSource:0}: Error finding container 57df00648526679e84aa214e291f0907b435ac1b35862ec837ed24b3d265dfe8: Status 404 returned error can't find the container with id 57df00648526679e84aa214e291f0907b435ac1b35862ec837ed24b3d265dfe8
Oct 30 13:17:51.181323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:51.180555    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"]
Oct 30 13:17:51.358061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:51.358023    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf" event={"ID":"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c","Type":"ContainerStarted","Data":"57df00648526679e84aa214e291f0907b435ac1b35862ec837ed24b3d265dfe8"}
Oct 30 13:17:51.435182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:51.435103    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:51.435424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:51.435403    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:51.436069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:51.436051    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:17:52.366018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:52.365980    4291 generic.go:356] "Generic (PLEG): container finished" podID="a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" containerID="bb0bcdb0a889548d52819ae49e63bef7b4bcab257ca079b35d012278ce185362" exitCode=0
Oct 30 13:17:52.366453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:52.366045    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf" event={"ID":"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c","Type":"ContainerDied","Data":"bb0bcdb0a889548d52819ae49e63bef7b4bcab257ca079b35d012278ce185362"}
Oct 30 13:17:53.538013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.537987    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:53.553234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.553206    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"]
Oct 30 13:17:53.553711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.553697    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" containerName="revoke-keys"
Oct 30 13:17:53.553760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.553713    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" containerName="revoke-keys"
Oct 30 13:17:53.553800 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.553788    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" containerName="revoke-keys"
Oct 30 13:17:53.556968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.556950    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.567353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.567323    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-kube-api-access-7v6vm\" (UniqueName: \"kubernetes.io/projected/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-kube-api-access-7v6vm\") pod \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") "
Oct 30 13:17:53.567491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.567408    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-xks-ca\") pod \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") "
Oct 30 13:17:53.567615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.567596    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-openshift-ca\") pod \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\" (UID: \"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c\") "
Oct 30 13:17:53.568248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.568210    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-openshift-ca" pod "a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" (UID: "a2a45093-dc1e-41f9-b7b1-b3d009c75d8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:17:53.569258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.569228    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:53.570322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.570296    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-xks-ca" pod "a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" (UID: "a2a45093-dc1e-41f9-b7b1-b3d009c75d8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:17:53.570532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.570500    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-kube-api-access-7v6vm" pod "a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" (UID: "a2a45093-dc1e-41f9-b7b1-b3d009c75d8c"). InnerVolumeSpecName "kube-api-access-7v6vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:17:53.578481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.578439    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"]
Oct 30 13:17:53.670443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670346    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670410    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670508    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670546    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670678    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670710    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqrv4\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-kube-api-access-gqrv4\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670737    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670856    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.670920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.670891    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.671061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.671002    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7v6vm\" (UniqueName: \"kubernetes.io/projected/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-kube-api-access-7v6vm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:53.671061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.671019    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:17:53.772474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772438    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772499    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772530    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqrv4\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-kube-api-access-gqrv4\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772551    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772578    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772594    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772718    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772780    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.772837    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.773152    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.773210    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.773278    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.773366    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.773724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.773369    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.775529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.775506    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.775684 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.775667    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.791712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.791681    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.792290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.792267    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqrv4\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-kube-api-access-gqrv4\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:53.875000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:53.874964    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:54.208253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.208219    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"]
Oct 30 13:17:54.210402 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:17:54.210368    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e213319_f5d9_4cb6_9ca5_cb6a38a4d218.slice/crio-1c649aa421c2c31f305602ef2aab79b14d6251b5bfb246619e6039fb08c297b2 WatchSource:0}: Error finding container 1c649aa421c2c31f305602ef2aab79b14d6251b5bfb246619e6039fb08c297b2: Status 404 returned error can't find the container with id 1c649aa421c2c31f305602ef2aab79b14d6251b5bfb246619e6039fb08c297b2
Oct 30 13:17:54.212888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.212852    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:54.212977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.212922    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:54.212977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.212953    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:17:54.383338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.383311    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"
Oct 30 13:17:54.383506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.383345    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf" event={"ID":"a2a45093-dc1e-41f9-b7b1-b3d009c75d8c","Type":"ContainerDied","Data":"57df00648526679e84aa214e291f0907b435ac1b35862ec837ed24b3d265dfe8"}
Oct 30 13:17:54.383506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.383400    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57df00648526679e84aa214e291f0907b435ac1b35862ec837ed24b3d265dfe8"
Oct 30 13:17:54.386293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.385078    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" event={"ID":"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218","Type":"ContainerStarted","Data":"9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be"}
Oct 30 13:17:54.386293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.385136    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" event={"ID":"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218","Type":"ContainerStarted","Data":"1c649aa421c2c31f305602ef2aab79b14d6251b5bfb246619e6039fb08c297b2"}
Oct 30 13:17:54.426744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.426685    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podStartSLOduration=1.426669624 podStartE2EDuration="1.426669624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:17:54.4257185 +0000 UTC m=+884.685956748" watchObservedRunningTime="2026-10-30 13:17:54.426669624 +0000 UTC m=+884.686907855"
Oct 30 13:17:54.875993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:54.875951    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:17:55.911656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:55.906701    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:17:55.911656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:55.906802    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:17:56.879659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:56.879048    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:17:56.879659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:56.879144    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:17:57.131721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.130273    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"]
Oct 30 13:17:57.160660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.159812    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"]
Oct 30 13:17:57.160660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.159961    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.177658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.175628    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:17:57.190780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.190751    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:17:57.191050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.191032    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:17:57.195658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.191229    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:17:57.288660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.279665    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.288660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.279719    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.288660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.279773    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrwf\" (UniqueName: \"kubernetes.io/projected/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-kube-api-access-8wrwf\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.415752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.415628    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.542059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.541922    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.544056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.542278    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrwf\" (UniqueName: \"kubernetes.io/projected/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-kube-api-access-8wrwf\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.544056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.428129    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.558197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.558160    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.597665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.596317    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrwf\" (UniqueName: \"kubernetes.io/projected/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-kube-api-access-8wrwf\") pod \"maas-api-revoke-sub-0bd0ad0d-xmw2f\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.679866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.679749    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:17:57.793664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.790838    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:17:57.854518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.854479    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:17:57.865658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.861706    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:57.889971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.889937    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:17:57.890252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.890223    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:17:57.933316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:57.933202    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:58.368802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:58.368743    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:58.636848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:58.636735    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:58.685834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:58.682899    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:17:58.685834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:58.683354    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:17:58.740660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:58.724718    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:58.827773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:58.827725    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.025815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.025711    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:17:59.026885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.026841    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:17:59.184710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.165772    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.185867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.185816    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:17:59.187191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.187152    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.188366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.188337    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:59.327721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.264959    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.334711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.265936    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:17:59.404489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.404458    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:17:59.438311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.438263    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.480695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.480664    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:17:59.696679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.674289    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"]
Oct 30 13:17:59.696679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.678189    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:17:59.696679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.678667    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:59.696679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.693986    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:17:59.696679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.694045    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:17:59.696679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.694078    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:17:59.696679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.694470    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:17:59.732318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.732288    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:17:59.790654 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.790606    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:17:59.837985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.835237    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:17:59.848478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.848447    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:17:59.884992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.884958    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:17:59.885245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:17:59.885220    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:18:00.034779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.032159    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:18:00.133789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.133752    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:18:00.876708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.876149    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:18:00.876708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.876224    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:18:00.927176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.927134    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"]
Oct 30 13:18:00.931337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.931316    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:00.953668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.949885    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"]
Oct 30 13:18:00.999842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.999807    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.000010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:00.999872    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd45h\" (UniqueName: \"kubernetes.io/projected/5a0a6daa-b398-4973-8c67-ac04fa3574e0-kube-api-access-rd45h\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.000054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.000012    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.048228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.048192    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f" event={"ID":"37fc1856-c68c-4da6-b6b4-928f13d1fdbe","Type":"ContainerStarted","Data":"00a159bd3b28660b7048727490f71cce19f32ce553d419899e16e919ae8b77e2"}
Oct 30 13:18:01.048442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.048249    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f" event={"ID":"37fc1856-c68c-4da6-b6b4-928f13d1fdbe","Type":"ContainerStarted","Data":"6450888d7cd0bb50497baf1072da9acd19afd4bf09feedfeaeeb697060a65125"}
Oct 30 13:18:01.088938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.088887    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f" podStartSLOduration=4.088870232 podStartE2EDuration="4.088870232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:17:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:01.086363264 +0000 UTC m=+891.346601499" watchObservedRunningTime="2026-10-30 13:18:01.088870232 +0000 UTC m=+891.349108465"
Oct 30 13:18:01.101230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.101194    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.101417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.101399    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd45h\" (UniqueName: \"kubernetes.io/projected/5a0a6daa-b398-4973-8c67-ac04fa3574e0-kube-api-access-rd45h\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.101539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.101526    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.102405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.102342    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.103819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.103794    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.121029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.121002    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd45h\" (UniqueName: \"kubernetes.io/projected/5a0a6daa-b398-4973-8c67-ac04fa3574e0-kube-api-access-rd45h\") pod \"maas-api-revoke-sub-ebda273e-hgl7k\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.246192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.246102    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:01.583517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.583480    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"]
Oct 30 13:18:01.584824 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:01.584795    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a0a6daa_b398_4973_8c67_ac04fa3574e0.slice/crio-6fd270c4a0d5821e383c5cd6c3195d1682993f3f304f983db85eea1acd4d0851 WatchSource:0}: Error finding container 6fd270c4a0d5821e383c5cd6c3195d1682993f3f304f983db85eea1acd4d0851: Status 404 returned error can't find the container with id 6fd270c4a0d5821e383c5cd6c3195d1682993f3f304f983db85eea1acd4d0851
Oct 30 13:18:01.875761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.875727    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body=
Oct 30 13:18:01.875929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:01.875801    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused"
Oct 30 13:18:02.059000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:02.058965    4291 generic.go:356] "Generic (PLEG): container finished" podID="37fc1856-c68c-4da6-b6b4-928f13d1fdbe" containerID="00a159bd3b28660b7048727490f71cce19f32ce553d419899e16e919ae8b77e2" exitCode=0
Oct 30 13:18:02.059430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:02.059060    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f" event={"ID":"37fc1856-c68c-4da6-b6b4-928f13d1fdbe","Type":"ContainerDied","Data":"00a159bd3b28660b7048727490f71cce19f32ce553d419899e16e919ae8b77e2"}
Oct 30 13:18:02.060747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:02.060721    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k" event={"ID":"5a0a6daa-b398-4973-8c67-ac04fa3574e0","Type":"ContainerStarted","Data":"5519af050eb1f78b62a5c85477b96482c57e5d3552f6acd63990d17d191ae64d"}
Oct 30 13:18:02.060863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:02.060769    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k" event={"ID":"5a0a6daa-b398-4973-8c67-ac04fa3574e0","Type":"ContainerStarted","Data":"6fd270c4a0d5821e383c5cd6c3195d1682993f3f304f983db85eea1acd4d0851"}
Oct 30 13:18:02.136913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:02.136797    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k" podStartSLOduration=2.13677547 podStartE2EDuration="2.13677547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:02.134469457 +0000 UTC m=+892.394707693" watchObservedRunningTime="2026-10-30 13:18:02.13677547 +0000 UTC m=+892.397013704"
Oct 30 13:18:03.070734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.070692    4291 generic.go:356] "Generic (PLEG): container finished" podID="5a0a6daa-b398-4973-8c67-ac04fa3574e0" containerID="5519af050eb1f78b62a5c85477b96482c57e5d3552f6acd63990d17d191ae64d" exitCode=0
Oct 30 13:18:03.071182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.070777    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k" event={"ID":"5a0a6daa-b398-4973-8c67-ac04fa3574e0","Type":"ContainerDied","Data":"5519af050eb1f78b62a5c85477b96482c57e5d3552f6acd63990d17d191ae64d"}
Oct 30 13:18:03.119331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.119243    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:18:03.120317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.120294    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:18:03.122626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.122606    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:18:03.228076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.228050    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:18:03.332612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.332574    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-xks-ca\") pod \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") "
Oct 30 13:18:03.332832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.332701    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-kube-api-access-8wrwf\" (UniqueName: \"kubernetes.io/projected/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-kube-api-access-8wrwf\") pod \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") "
Oct 30 13:18:03.333991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.333945    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-openshift-ca\") pod \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\" (UID: \"37fc1856-c68c-4da6-b6b4-928f13d1fdbe\") "
Oct 30 13:18:03.335097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.335065    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-openshift-ca" pod "37fc1856-c68c-4da6-b6b4-928f13d1fdbe" (UID: "37fc1856-c68c-4da6-b6b4-928f13d1fdbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:03.339783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.338314    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-xks-ca" pod "37fc1856-c68c-4da6-b6b4-928f13d1fdbe" (UID: "37fc1856-c68c-4da6-b6b4-928f13d1fdbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:03.342056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.341010    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-kube-api-access-8wrwf" pod "37fc1856-c68c-4da6-b6b4-928f13d1fdbe" (UID: "37fc1856-c68c-4da6-b6b4-928f13d1fdbe"). InnerVolumeSpecName "kube-api-access-8wrwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:03.436669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.436554    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:03.436669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.436585    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:03.436669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:03.436595    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wrwf\" (UniqueName: \"kubernetes.io/projected/37fc1856-c68c-4da6-b6b4-928f13d1fdbe-kube-api-access-8wrwf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:04.078538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.078506    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"
Oct 30 13:18:04.079054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.078545    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f" event={"ID":"37fc1856-c68c-4da6-b6b4-928f13d1fdbe","Type":"ContainerDied","Data":"6450888d7cd0bb50497baf1072da9acd19afd4bf09feedfeaeeb697060a65125"}
Oct 30 13:18:04.079054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.078599    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6450888d7cd0bb50497baf1072da9acd19afd4bf09feedfeaeeb697060a65125"
Oct 30 13:18:04.242227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.242195    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:04.347155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.347119    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a0a6daa-b398-4973-8c67-ac04fa3574e0-kube-api-access-rd45h\" (UniqueName: \"kubernetes.io/projected/5a0a6daa-b398-4973-8c67-ac04fa3574e0-kube-api-access-rd45h\") pod \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") "
Oct 30 13:18:04.347327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.347206    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-openshift-ca\") pod \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") "
Oct 30 13:18:04.347375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.347351    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-xks-ca\") pod \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\" (UID: \"5a0a6daa-b398-4973-8c67-ac04fa3574e0\") "
Oct 30 13:18:04.347663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.347605    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-openshift-ca" pod "5a0a6daa-b398-4973-8c67-ac04fa3574e0" (UID: "5a0a6daa-b398-4973-8c67-ac04fa3574e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:04.347938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.347921    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:04.349431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.349406    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-xks-ca" pod "5a0a6daa-b398-4973-8c67-ac04fa3574e0" (UID: "5a0a6daa-b398-4973-8c67-ac04fa3574e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:04.349553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.349532    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a0a6daa-b398-4973-8c67-ac04fa3574e0-kube-api-access-rd45h" pod "5a0a6daa-b398-4973-8c67-ac04fa3574e0" (UID: "5a0a6daa-b398-4973-8c67-ac04fa3574e0"). InnerVolumeSpecName "kube-api-access-rd45h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:04.448719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.448600    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a0a6daa-b398-4973-8c67-ac04fa3574e0-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:04.448719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:04.448665    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd45h\" (UniqueName: \"kubernetes.io/projected/5a0a6daa-b398-4973-8c67-ac04fa3574e0-kube-api-access-rd45h\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:05.085984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.085958    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"
Oct 30 13:18:05.086392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.085992    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k" event={"ID":"5a0a6daa-b398-4973-8c67-ac04fa3574e0","Type":"ContainerDied","Data":"6fd270c4a0d5821e383c5cd6c3195d1682993f3f304f983db85eea1acd4d0851"}
Oct 30 13:18:05.086392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.086040    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6fd270c4a0d5821e383c5cd6c3195d1682993f3f304f983db85eea1acd4d0851"
Oct 30 13:18:05.148138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.148104    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"]
Oct 30 13:18:05.148566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.148553    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5a0a6daa-b398-4973-8c67-ac04fa3574e0" containerName="revoke-keys"
Oct 30 13:18:05.148627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.148567    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a0a6daa-b398-4973-8c67-ac04fa3574e0" containerName="revoke-keys"
Oct 30 13:18:05.148627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.148590    4291 cpu_manager.go:414] "containerMap: removing container" podUID="37fc1856-c68c-4da6-b6b4-928f13d1fdbe" containerName="revoke-keys"
Oct 30 13:18:05.148627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.148594    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37fc1856-c68c-4da6-b6b4-928f13d1fdbe" containerName="revoke-keys"
Oct 30 13:18:05.148731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.148690    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a0a6daa-b398-4973-8c67-ac04fa3574e0" containerName="revoke-keys"
Oct 30 13:18:05.148731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.148702    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="37fc1856-c68c-4da6-b6b4-928f13d1fdbe" containerName="revoke-keys"
Oct 30 13:18:05.152227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.152211    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.166902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.166869    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:05.167158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.167137    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:18:05.167256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.167147    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:05.167256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.167147    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:05.169212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.169191    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"]
Oct 30 13:18:05.259217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.259179    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48c72\" (UniqueName: \"kubernetes.io/projected/7315958d-a84f-4082-a926-e7474762d355-kube-api-access-48c72\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.259416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.259391    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7315958d-a84f-4082-a926-e7474762d355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.259479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.259441    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7315958d-a84f-4082-a926-e7474762d355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.360471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.360431    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7315958d-a84f-4082-a926-e7474762d355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.360629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.360482    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7315958d-a84f-4082-a926-e7474762d355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.360629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.360543    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48c72\" (UniqueName: \"kubernetes.io/projected/7315958d-a84f-4082-a926-e7474762d355-kube-api-access-48c72\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.361292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.361273    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7315958d-a84f-4082-a926-e7474762d355-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.363016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.362997    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7315958d-a84f-4082-a926-e7474762d355-maas-api-xks-ca\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.380125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.380094    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48c72\" (UniqueName: \"kubernetes.io/projected/7315958d-a84f-4082-a926-e7474762d355-kube-api-access-48c72\") pod \"maas-api-revoke-sub-117e277a-kxx2k\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.465813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.465777    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:05.775100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:05.775072    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"]
Oct 30 13:18:05.775355 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:05.775333    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7315958d_a84f_4082_a926_e7474762d355.slice/crio-ee24b34cd26e6098055123b86961dbe74522e518db6b174a7a6f875ddf7480fe WatchSource:0}: Error finding container ee24b34cd26e6098055123b86961dbe74522e518db6b174a7a6f875ddf7480fe: Status 404 returned error can't find the container with id ee24b34cd26e6098055123b86961dbe74522e518db6b174a7a6f875ddf7480fe
Oct 30 13:18:06.094731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:06.094696    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k" event={"ID":"7315958d-a84f-4082-a926-e7474762d355","Type":"ContainerStarted","Data":"21e898b8ee1cbebae95c304c61f43cf4d6c766660d545f01d088d81c6abe1495"}
Oct 30 13:18:06.095097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:06.094749    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k" event={"ID":"7315958d-a84f-4082-a926-e7474762d355","Type":"ContainerStarted","Data":"ee24b34cd26e6098055123b86961dbe74522e518db6b174a7a6f875ddf7480fe"}
Oct 30 13:18:06.134394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:06.134345    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k" podStartSLOduration=1.134330237 podStartE2EDuration="1.134330237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:06.131059221 +0000 UTC m=+896.391297455" watchObservedRunningTime="2026-10-30 13:18:06.134330237 +0000 UTC m=+896.394568470"
Oct 30 13:18:07.102372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:07.102342    4291 generic.go:356] "Generic (PLEG): container finished" podID="7315958d-a84f-4082-a926-e7474762d355" containerID="21e898b8ee1cbebae95c304c61f43cf4d6c766660d545f01d088d81c6abe1495" exitCode=0
Oct 30 13:18:07.102830 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:07.102414    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k" event={"ID":"7315958d-a84f-4082-a926-e7474762d355","Type":"ContainerDied","Data":"21e898b8ee1cbebae95c304c61f43cf4d6c766660d545f01d088d81c6abe1495"}
Oct 30 13:18:08.217650 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:18:08.217599    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:18:08.257362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.257335    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:08.294363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.294322    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7315958d-a84f-4082-a926-e7474762d355-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7315958d-a84f-4082-a926-e7474762d355-maas-api-openshift-ca\") pod \"7315958d-a84f-4082-a926-e7474762d355\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") "
Oct 30 13:18:08.294558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.294388    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7315958d-a84f-4082-a926-e7474762d355-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7315958d-a84f-4082-a926-e7474762d355-maas-api-xks-ca\") pod \"7315958d-a84f-4082-a926-e7474762d355\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") "
Oct 30 13:18:08.294558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.294525    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7315958d-a84f-4082-a926-e7474762d355-kube-api-access-48c72\" (UniqueName: \"kubernetes.io/projected/7315958d-a84f-4082-a926-e7474762d355-kube-api-access-48c72\") pod \"7315958d-a84f-4082-a926-e7474762d355\" (UID: \"7315958d-a84f-4082-a926-e7474762d355\") "
Oct 30 13:18:08.294816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.294786    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7315958d-a84f-4082-a926-e7474762d355-maas-api-openshift-ca" pod "7315958d-a84f-4082-a926-e7474762d355" (UID: "7315958d-a84f-4082-a926-e7474762d355"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:08.295210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.295119    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7315958d-a84f-4082-a926-e7474762d355-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:08.297432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.297406    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7315958d-a84f-4082-a926-e7474762d355-kube-api-access-48c72" pod "7315958d-a84f-4082-a926-e7474762d355" (UID: "7315958d-a84f-4082-a926-e7474762d355"). InnerVolumeSpecName "kube-api-access-48c72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:08.297528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.297451    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7315958d-a84f-4082-a926-e7474762d355-maas-api-xks-ca" pod "7315958d-a84f-4082-a926-e7474762d355" (UID: "7315958d-a84f-4082-a926-e7474762d355"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:08.396853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.396761    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48c72\" (UniqueName: \"kubernetes.io/projected/7315958d-a84f-4082-a926-e7474762d355-kube-api-access-48c72\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:08.396853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:08.396793    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7315958d-a84f-4082-a926-e7474762d355-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:09.117691 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:09.117656    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"
Oct 30 13:18:09.117691 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:09.117684    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k" event={"ID":"7315958d-a84f-4082-a926-e7474762d355","Type":"ContainerDied","Data":"ee24b34cd26e6098055123b86961dbe74522e518db6b174a7a6f875ddf7480fe"}
Oct 30 13:18:09.117881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:09.117733    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee24b34cd26e6098055123b86961dbe74522e518db6b174a7a6f875ddf7480fe"
Oct 30 13:18:14.204701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.204658    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"]
Oct 30 13:18:14.205152 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.205138    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7315958d-a84f-4082-a926-e7474762d355" containerName="revoke-keys"
Oct 30 13:18:14.205193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.205153    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7315958d-a84f-4082-a926-e7474762d355" containerName="revoke-keys"
Oct 30 13:18:14.205255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.205245    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7315958d-a84f-4082-a926-e7474762d355" containerName="revoke-keys"
Oct 30 13:18:14.210163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.210141    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.224571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.224540    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"]
Oct 30 13:18:14.232366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.232324    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:14.232366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.232344    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:14.232366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.232324    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:14.236372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.236347    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:18:14.265990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.265954    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.266150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.266095    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.266194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.266170    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49mfd\" (UniqueName: \"kubernetes.io/projected/79e878c2-d9c0-4434-addc-04fbc0894c89-kube-api-access-49mfd\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.367927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.367876    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.368095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.367976    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49mfd\" (UniqueName: \"kubernetes.io/projected/79e878c2-d9c0-4434-addc-04fbc0894c89-kube-api-access-49mfd\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.368138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.368112    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.368789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.368770    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.370447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.370421    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-xks-ca\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.387533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.387498    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49mfd\" (UniqueName: \"kubernetes.io/projected/79e878c2-d9c0-4434-addc-04fbc0894c89-kube-api-access-49mfd\") pod \"maas-api-revoke-sub-27de93c7-5299z\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.524710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.524594    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:14.831096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:14.831065    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"]
Oct 30 13:18:14.831757 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:14.831724    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e878c2_d9c0_4434_addc_04fbc0894c89.slice/crio-46f23a0bfd8ef109d447bcc5740d4d838be77534cadcea9bc4e33390273120ed WatchSource:0}: Error finding container 46f23a0bfd8ef109d447bcc5740d4d838be77534cadcea9bc4e33390273120ed: Status 404 returned error can't find the container with id 46f23a0bfd8ef109d447bcc5740d4d838be77534cadcea9bc4e33390273120ed
Oct 30 13:18:15.169083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:15.169049    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z" event={"ID":"79e878c2-d9c0-4434-addc-04fbc0894c89","Type":"ContainerStarted","Data":"46f23a0bfd8ef109d447bcc5740d4d838be77534cadcea9bc4e33390273120ed"}
Oct 30 13:18:16.177065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:16.177029    4291 generic.go:356] "Generic (PLEG): container finished" podID="79e878c2-d9c0-4434-addc-04fbc0894c89" containerID="7bfb904866b1db1a750b079474d8053e750c66f33daafe8abb31e4a1d76908b2" exitCode=0
Oct 30 13:18:16.177462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:16.177114    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z" event={"ID":"79e878c2-d9c0-4434-addc-04fbc0894c89","Type":"ContainerDied","Data":"7bfb904866b1db1a750b079474d8053e750c66f33daafe8abb31e4a1d76908b2"}
Oct 30 13:18:16.605824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:16.605791    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:18:17.410833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.403778    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:17.509285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.509171    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-openshift-ca\") pod \"79e878c2-d9c0-4434-addc-04fbc0894c89\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") "
Oct 30 13:18:17.509285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.509259    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79e878c2-d9c0-4434-addc-04fbc0894c89-kube-api-access-49mfd\" (UniqueName: \"kubernetes.io/projected/79e878c2-d9c0-4434-addc-04fbc0894c89-kube-api-access-49mfd\") pod \"79e878c2-d9c0-4434-addc-04fbc0894c89\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") "
Oct 30 13:18:17.509533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.509362    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-xks-ca\") pod \"79e878c2-d9c0-4434-addc-04fbc0894c89\" (UID: \"79e878c2-d9c0-4434-addc-04fbc0894c89\") "
Oct 30 13:18:17.509675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.509623    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-openshift-ca" pod "79e878c2-d9c0-4434-addc-04fbc0894c89" (UID: "79e878c2-d9c0-4434-addc-04fbc0894c89"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:17.510286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.510263    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:17.512071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.512042    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e878c2-d9c0-4434-addc-04fbc0894c89-kube-api-access-49mfd" pod "79e878c2-d9c0-4434-addc-04fbc0894c89" (UID: "79e878c2-d9c0-4434-addc-04fbc0894c89"). InnerVolumeSpecName "kube-api-access-49mfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:17.512167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.512090    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-xks-ca" pod "79e878c2-d9c0-4434-addc-04fbc0894c89" (UID: "79e878c2-d9c0-4434-addc-04fbc0894c89"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:17.618659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.613621    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49mfd\" (UniqueName: \"kubernetes.io/projected/79e878c2-d9c0-4434-addc-04fbc0894c89-kube-api-access-49mfd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:17.618659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:17.613681    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79e878c2-d9c0-4434-addc-04fbc0894c89-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:18.247947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:18.236899    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"
Oct 30 13:18:18.247947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:18.247771    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z" event={"ID":"79e878c2-d9c0-4434-addc-04fbc0894c89","Type":"ContainerDied","Data":"46f23a0bfd8ef109d447bcc5740d4d838be77534cadcea9bc4e33390273120ed"}
Oct 30 13:18:18.247947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:18.247854    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46f23a0bfd8ef109d447bcc5740d4d838be77534cadcea9bc4e33390273120ed"
Oct 30 13:18:19.575660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:19.574757    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:18:19.575660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:19.574864    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:19.713660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:19.708732    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.732920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:20.731727    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:20.775012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:20.774945    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:21.697148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:21.697087    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:23.337882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.337845    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"]
Oct 30 13:18:23.338522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.338501    4291 cpu_manager.go:414] "containerMap: removing container" podUID="79e878c2-d9c0-4434-addc-04fbc0894c89" containerName="revoke-keys"
Oct 30 13:18:23.338574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.338525    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79e878c2-d9c0-4434-addc-04fbc0894c89" containerName="revoke-keys"
Oct 30 13:18:23.338736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.338723    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="79e878c2-d9c0-4434-addc-04fbc0894c89" containerName="revoke-keys"
Oct 30 13:18:23.345272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.345248    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.366896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.366869    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"]
Oct 30 13:18:23.433430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433397    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433443    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433461    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2jmf\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-kube-api-access-q2jmf\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433602    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb73da3-d431-4205-9dce-234b12384bb5-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433671    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433751    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433777    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433801    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb73da3-d431-4205-9dce-234b12384bb5-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.433849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.433832    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534448    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534500    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534516    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2jmf\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-kube-api-access-q2jmf\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534569    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb73da3-d431-4205-9dce-234b12384bb5-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534603    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534630    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534680    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534712    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb73da3-d431-4205-9dce-234b12384bb5-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534742    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.534967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.534945    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.535174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.535153    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.535227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.535184    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.535296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.535273    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.535688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.535659    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb73da3-d431-4205-9dce-234b12384bb5-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.537127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.537108    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.537451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.537433    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb73da3-d431-4205-9dce-234b12384bb5-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.553550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.553523    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2jmf\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-kube-api-access-q2jmf\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.553844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.553822    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.665818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.665789    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:23.993239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.993203    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"]
Oct 30 13:18:23.994009 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:23.993984    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb73da3_d431_4205_9dce_234b12384bb5.slice/crio-d649b94383782fd54ed154a7faf64de925f66d7e6cac24251f4df14167a11e88 WatchSource:0}: Error finding container d649b94383782fd54ed154a7faf64de925f66d7e6cac24251f4df14167a11e88: Status 404 returned error can't find the container with id d649b94383782fd54ed154a7faf64de925f66d7e6cac24251f4df14167a11e88
Oct 30 13:18:23.996445 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.996418    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:23.996523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.996478    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:23.996523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:23.996509    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:24.433580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:24.433538    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" event={"ID":"9cb73da3-d431-4205-9dce-234b12384bb5","Type":"ContainerStarted","Data":"514a23ba4d8550e36f82910d7d3a5a86fe6fad422db32bfb4d19a69889ea574b"}
Oct 30 13:18:24.433976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:24.433607    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" event={"ID":"9cb73da3-d431-4205-9dce-234b12384bb5","Type":"ContainerStarted","Data":"d649b94383782fd54ed154a7faf64de925f66d7e6cac24251f4df14167a11e88"}
Oct 30 13:18:24.485449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:24.485393    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podStartSLOduration=1.4853782739999999 podStartE2EDuration="1.485378274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:24.47999332 +0000 UTC m=+914.740231577" watchObservedRunningTime="2026-10-30 13:18:24.485378274 +0000 UTC m=+914.745616507"
Oct 30 13:18:24.666867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:24.666829    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:25.282834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.282798    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"]
Oct 30 13:18:25.288324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.288287    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"]
Oct 30 13:18:25.288492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.288294    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.319136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.314791    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:25.319136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.314818    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:25.319136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.314818    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:18:25.319136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.314870    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:25.386714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.383389    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.386714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.383546    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.386714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.383710    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvxw2\" (UniqueName: \"kubernetes.io/projected/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-kube-api-access-cvxw2\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.485660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.485468    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.485660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.485665    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvxw2\" (UniqueName: \"kubernetes.io/projected/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-kube-api-access-cvxw2\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.486230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.485718    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.489246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.489129    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.496108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.496072    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.518519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.518486    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvxw2\" (UniqueName: \"kubernetes.io/projected/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-kube-api-access-cvxw2\") pod \"maas-api-revoke-sub-e03fc693-nq6wg\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.608677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.606900    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:25.683696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.676803    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:18:25.683696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:25.677185    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:18:26.019974 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:26.019941    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0afc7bf6_8c6a_4f09_9f22_7d629cfe32f5.slice/crio-7f729803da3decfd18913a20f684e2b561c2c21c1194372d98b0613e4da4553f WatchSource:0}: Error finding container 7f729803da3decfd18913a20f684e2b561c2c21c1194372d98b0613e4da4553f: Status 404 returned error can't find the container with id 7f729803da3decfd18913a20f684e2b561c2c21c1194372d98b0613e4da4553f
Oct 30 13:18:26.021605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:26.021577    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"]
Oct 30 13:18:26.449060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:26.449021    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg" event={"ID":"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5","Type":"ContainerStarted","Data":"7f729803da3decfd18913a20f684e2b561c2c21c1194372d98b0613e4da4553f"}
Oct 30 13:18:26.674660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:26.673998    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:18:26.674660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:26.674057    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:18:27.479819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:27.479615    4291 generic.go:356] "Generic (PLEG): container finished" podID="0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" containerID="dc415f5f3a7d398aa07f4418d5af1804351979007f2465873373a08f92c8701f" exitCode=0
Oct 30 13:18:27.479819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:27.479679    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg" event={"ID":"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5","Type":"ContainerDied","Data":"dc415f5f3a7d398aa07f4418d5af1804351979007f2465873373a08f92c8701f"}
Oct 30 13:18:27.674810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:27.674753    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:18:27.675225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:27.674854    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:18:28.306621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.306228    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"]
Oct 30 13:18:28.315345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.315313    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.327299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.327274    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"]
Oct 30 13:18:28.442909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.442873    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.442947    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.443033    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.443094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vcmz\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-kube-api-access-5vcmz\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.443157    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7d1006-839a-487b-89d2-5155bac0c5f1-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.443197    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.443253    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.443289    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.443375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.443364    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546004    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546087    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546144    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546224    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546273    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546338    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vcmz\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-kube-api-access-5vcmz\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546369    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7d1006-839a-487b-89d2-5155bac0c5f1-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546410    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546463    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.546971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.546941    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.547231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.547208    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.547449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.547431    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.548189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.548164    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.548543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.548518    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7d1006-839a-487b-89d2-5155bac0c5f1-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.583697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.581757    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.583697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.582308    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.585001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.584978    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.586692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.586669    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vcmz\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-kube-api-access-5vcmz\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.632165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.632138    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:28.672672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.662877    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:28.676220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.676024    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:18:28.676220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.676084    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:18:28.748550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.748522    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-openshift-ca\") pod \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") "
Oct 30 13:18:28.748701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.748692    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-kube-api-access-cvxw2\" (UniqueName: \"kubernetes.io/projected/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-kube-api-access-cvxw2\") pod \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") "
Oct 30 13:18:28.748754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.748728    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-xks-ca\") pod \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\" (UID: \"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5\") "
Oct 30 13:18:28.748978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.748948    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-openshift-ca" pod "0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" (UID: "0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:28.749267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.749248    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:28.775626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.775591    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-kube-api-access-cvxw2" pod "0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" (UID: "0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5"). InnerVolumeSpecName "kube-api-access-cvxw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:28.778809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.778778    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-xks-ca" pod "0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" (UID: "0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:28.881678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.874205    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvxw2\" (UniqueName: \"kubernetes.io/projected/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-kube-api-access-cvxw2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:28.881678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.874246    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:28.988995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.988970    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"]
Oct 30 13:18:28.989616 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:28.989587    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde7d1006_839a_487b_89d2_5155bac0c5f1.slice/crio-5a95c413bfb2ca1259beefa20b0372701fa56c0fb5dc8a52dc3458d8b2531fae WatchSource:0}: Error finding container 5a95c413bfb2ca1259beefa20b0372701fa56c0fb5dc8a52dc3458d8b2531fae: Status 404 returned error can't find the container with id 5a95c413bfb2ca1259beefa20b0372701fa56c0fb5dc8a52dc3458d8b2531fae
Oct 30 13:18:28.992556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.992527    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:28.992652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.992589    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:28.992652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:28.992615    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:29.494504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.494468    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg" event={"ID":"0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5","Type":"ContainerDied","Data":"7f729803da3decfd18913a20f684e2b561c2c21c1194372d98b0613e4da4553f"}
Oct 30 13:18:29.494773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.494523    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f729803da3decfd18913a20f684e2b561c2c21c1194372d98b0613e4da4553f"
Oct 30 13:18:29.494773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.494476    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"
Oct 30 13:18:29.496357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.496331    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" event={"ID":"de7d1006-839a-487b-89d2-5155bac0c5f1","Type":"ContainerStarted","Data":"61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d"}
Oct 30 13:18:29.496466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.496372    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" event={"ID":"de7d1006-839a-487b-89d2-5155bac0c5f1","Type":"ContainerStarted","Data":"5a95c413bfb2ca1259beefa20b0372701fa56c0fb5dc8a52dc3458d8b2531fae"}
Oct 30 13:18:29.549089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.549022    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podStartSLOduration=1.548993715 podStartE2EDuration="1.548993715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:29.539377587 +0000 UTC m=+919.799615822" watchObservedRunningTime="2026-10-30 13:18:29.548993715 +0000 UTC m=+919.809231977"
Oct 30 13:18:29.633124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.633087    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:29.667518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.667094    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:18:29.667518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:29.667155    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:18:30.634445 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:30.634407    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:18:30.634950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:30.634463    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:18:30.668663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:30.667987    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:18:30.668663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:30.668049    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:18:31.633212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:31.633170    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:18:31.633406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:31.633241    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:18:32.022800 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:32.022702    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:32.023279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:32.023156    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:32.024203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:32.024178    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:18:32.632537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:32.632494    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:18:32.632743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:32.632549    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:18:33.633486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:33.633436    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:18:33.633968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:33.633504    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:18:34.632752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:34.632710    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:18:34.632937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:34.632767    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:18:35.633506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.633473    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body=
Oct 30 13:18:35.634010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.633526    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused"
Oct 30 13:18:35.765991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.765959    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"]
Oct 30 13:18:35.766433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.766420    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" containerName="revoke-keys"
Oct 30 13:18:35.766474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.766434    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" containerName="revoke-keys"
Oct 30 13:18:35.766547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.766538    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" containerName="revoke-keys"
Oct 30 13:18:35.770335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.770314    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812155    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca4d001-f934-405c-b02d-16e95de2cb55-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812217    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812296    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812328    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812353    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812387    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smgsv\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-kube-api-access-smgsv\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812421    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812448    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.828700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.812522    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.844046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.843933    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"]
Oct 30 13:18:35.889282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.889193    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"]
Oct 30 13:18:35.894201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.894180    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:35.898383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.898127    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"]
Oct 30 13:18:35.913526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.913493    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.913671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.913554    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.913671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.913643    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.913797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.913784    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca4d001-f934-405c-b02d-16e95de2cb55-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.913850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.913829    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.913942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.913913    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.914180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.914160    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.915903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.914755    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.916011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.914863    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.916011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.915108    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.916011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.915109    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.916011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.915879    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.916011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.915989    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smgsv\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-kube-api-access-smgsv\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.916268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.916100    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca4d001-f934-405c-b02d-16e95de2cb55-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.919071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.919048    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.919167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.919098    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.936556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.936526    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:35.938324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:35.938300    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smgsv\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-kube-api-access-smgsv\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:36.017465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.017348    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.017684 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.017510    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.017811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.017741    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bw27l\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-kube-api-access-bw27l\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.017940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.017919    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.018034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.017953    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.018034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.017999    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.018160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.018131    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.018322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.018282    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.018380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.018327    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.037767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.037731    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"]
Oct 30 13:18:36.042969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.042950    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.061897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.061863    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"]
Oct 30 13:18:36.085741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.085712    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:36.176869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.174617    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6b7b93a3\""
Oct 30 13:18:36.177381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.177335    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.177951    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178094    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178147    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178226    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47c1e82-2cff-4c8d-b702-05da93c3ba41-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178277    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178306    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bw27l\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-kube-api-access-bw27l\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178460    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178509    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178579    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178609    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdlp6\" (UniqueName: \"kubernetes.io/projected/e47c1e82-2cff-4c8d-b702-05da93c3ba41-kube-api-access-hdlp6\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.178679    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.179025    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-certs\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.180184    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-data\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.182610    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6b7b93a3-dockercfg-cx4q9\""
Oct 30 13:18:36.189659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.188249    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istiod-ca-cert\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.195086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.195047    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-credential-socket\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.207553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.207150    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-envoy\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.207553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.207497    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bw27l\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-kube-api-access-bw27l\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.207895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.207865    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-podinfo\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.210088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.210038    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-token\") pod \"e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") " pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.281067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.280118    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdlp6\" (UniqueName: \"kubernetes.io/projected/e47c1e82-2cff-4c8d-b702-05da93c3ba41-kube-api-access-hdlp6\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.281067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.280341    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47c1e82-2cff-4c8d-b702-05da93c3ba41-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.282200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.282135    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47c1e82-2cff-4c8d-b702-05da93c3ba41-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.310282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.309348    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdlp6\" (UniqueName: \"kubernetes.io/projected/e47c1e82-2cff-4c8d-b702-05da93c3ba41-kube-api-access-hdlp6\") pod \"payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.360878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.360842    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:36.509067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.508975    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:36.539135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.539096    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"]
Oct 30 13:18:36.542679 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:36.542645    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca4d001_f934_405c_b02d_16e95de2cb55.slice/crio-c194b06e3148452fa31de1e1034fe25b9509fcebd750b55abc72757e3aaba377 WatchSource:0}: Error finding container c194b06e3148452fa31de1e1034fe25b9509fcebd750b55abc72757e3aaba377: Status 404 returned error can't find the container with id c194b06e3148452fa31de1e1034fe25b9509fcebd750b55abc72757e3aaba377
Oct 30 13:18:36.545506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.545466    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.545617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.545560    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.545617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.545609    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.623981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.623904    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" event={"ID":"7ca4d001-f934-405c-b02d-16e95de2cb55","Type":"ContainerStarted","Data":"c194b06e3148452fa31de1e1034fe25b9509fcebd750b55abc72757e3aaba377"}
Oct 30 13:18:36.676422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.676274    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"]
Oct 30 13:18:36.678295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.678272    4291 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 13:18:36.689460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.689436    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:36.691022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.690998    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:36.691693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.691674    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 13:18:36.827549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.827434    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"]
Oct 30 13:18:36.838058 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:36.838033    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02d3ae08_43a5_4453_b9ba_b1da9eb2d41d.slice/crio-c2e27ecd8f2d6251a0710534e842cc728bfd2c093ba71292d1b0c943f49d935a WatchSource:0}: Error finding container c2e27ecd8f2d6251a0710534e842cc728bfd2c093ba71292d1b0c943f49d935a: Status 404 returned error can't find the container with id c2e27ecd8f2d6251a0710534e842cc728bfd2c093ba71292d1b0c943f49d935a
Oct 30 13:18:36.840152 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.840115    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.840302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.840200    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:36.840302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:36.840244    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:37.635135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:37.635096    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" event={"ID":"e47c1e82-2cff-4c8d-b702-05da93c3ba41","Type":"ContainerStarted","Data":"afe891e219deebdeb129a8b7bc19a2b90870f65f8e106ebdad03450152996255"}
Oct 30 13:18:37.637773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:37.637717    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" event={"ID":"7ca4d001-f934-405c-b02d-16e95de2cb55","Type":"ContainerStarted","Data":"c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0"}
Oct 30 13:18:37.641264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:37.641236    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" event={"ID":"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d","Type":"ContainerStarted","Data":"0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645"}
Oct 30 13:18:37.641361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:37.641283    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" event={"ID":"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d","Type":"ContainerStarted","Data":"c2e27ecd8f2d6251a0710534e842cc728bfd2c093ba71292d1b0c943f49d935a"}
Oct 30 13:18:37.707467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:37.707404    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podStartSLOduration=2.707386703 podStartE2EDuration="2.707386703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:37.705058814 +0000 UTC m=+927.965297063" watchObservedRunningTime="2026-10-30 13:18:37.707386703 +0000 UTC m=+927.967624938"
Oct 30 13:18:37.785138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:37.773455    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podStartSLOduration=2.773433152 podStartE2EDuration="2.773433152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:37.749434345 +0000 UTC m=+928.009672578" watchObservedRunningTime="2026-10-30 13:18:37.773433152 +0000 UTC m=+928.033671388"
Oct 30 13:18:38.093878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:38.093783    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:38.095825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:38.095788    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:38.095972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:38.095842    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:38.453184 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:18:38.451855    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:18:38.522660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:38.518681    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:38.522660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:38.520152    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:38.522660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:38.520202    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:39.099885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.073392    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"]
Oct 30 13:18:39.099885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.093796    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:39.099885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.093871    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:39.251658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.238060    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.292609    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.317988    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.318039    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.318128    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.318208    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f257e951-f407-4d7f-baaf-24f7327f21d0-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.318247    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggrkv\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-kube-api-access-ggrkv\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.318345    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.318435    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.327666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.318506    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.367114 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.367009    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"]
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.421590    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.421667    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.421738    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.421790    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f257e951-f407-4d7f-baaf-24f7327f21d0-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.421816    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggrkv\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-kube-api-access-ggrkv\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.421890    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.421963    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.422020    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.424660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.422111    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.430659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.430454    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f257e951-f407-4d7f-baaf-24f7327f21d0-istiod-ca-cert\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.430857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.430778    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-data\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.452707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.434469    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.452707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.434779    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-certs\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.452707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.435046    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-credential-socket\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.452707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.435296    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-envoy\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.461657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.455619    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggrkv\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-kube-api-access-ggrkv\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.473687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.467698    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-token\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.490659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.488391    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-podinfo\") pod \"e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") " pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.514660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.514103    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:39.514660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.514163    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:39.691732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.679910    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:39.691732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.680272    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:39.701664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.699976    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:18:39.701664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.701093    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:39.733674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.729767    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:39.735872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.735822    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:39.815662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.810727    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:39.866889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.866784    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 13:18:39.974850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:39.974736    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:40.004660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.003729    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 13:18:40.096997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.096958    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:40.097260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.097229    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:40.523988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.510993    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:40.523988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.511063    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:40.704659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.681758    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:40.704659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.682231    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:18:40.833770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.820979    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:40.833770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.821442    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:18:40.833770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.822342    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:40.833770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.823782    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 13:18:40.833770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.824134    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:18:40.833770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.824189    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:18:40.872676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.847582    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:18:40.872676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.847975    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:18:40.872676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.865825    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:18:40.872676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:40.865901    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:18:41.015660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.014836    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 13:18:41.031882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.015908    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": context deadline exceeded"
Oct 30 13:18:41.031882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.017427    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:18:41.031882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.017488    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:18:41.140660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.137047    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:41.191192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.144706    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.191713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.191052    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:41.199659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.191907    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:18:41.539765 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.539732    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"]
Oct 30 13:18:41.572555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.572508    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:41.626660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.604562    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:41.626660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.604625    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:41.626660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.602786    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:41.626660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.605218    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:41.729709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.705117    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.869127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.851470    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.869127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.860168    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:18:41.869127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.869110    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:41.869975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.869946    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 13:18:41.870352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:41.870329    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": context deadline exceeded"
Oct 30 13:18:42.031730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.017388    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:42.031730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.017764    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 13:18:42.031730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.025493    4291 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/health\": context deadline exceeded"
Oct 30 13:18:42.089491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.089456    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:42.123101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.123046    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:42.139009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.138977    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 13:18:42.139242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.139227    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:18:42.202778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.202731    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 13:18:42.493065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.493016    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" event={"ID":"f257e951-f407-4d7f-baaf-24f7327f21d0","Type":"ContainerStarted","Data":"6faa5ff41b13bea53f81ba3111abba4e19636608966ddf6918b5e6ca4ab3f5ba"}
Oct 30 13:18:42.526111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.526074    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:42.526415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.526381    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:42.788658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.779322    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:18:42.863666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.856512    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:42.902204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.863920    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:42.902204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.873187    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 13:18:42.902204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:42.873555    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.112659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.083075    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:43.115906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.094881    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:43.116156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.116112    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:43.223724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.209779    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:43.473447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.472726    4291 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:43.743666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.727762    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:43.743666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.727833    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:43.830708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.830057    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" event={"ID":"f257e951-f407-4d7f-baaf-24f7327f21d0","Type":"ContainerStarted","Data":"90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208"}
Oct 30 13:18:43.845675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.842735    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:43.926971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.926928    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.928304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.928274    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": context deadline exceeded"
Oct 30 13:18:43.949070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.948117    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:18:44.074938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.950195    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": context deadline exceeded"
Oct 30 13:18:44.074938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:43.973818    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podStartSLOduration=5.973797564 podStartE2EDuration="5.973797564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:43.972204546 +0000 UTC m=+934.232442785" watchObservedRunningTime="2026-10-30 13:18:43.973797564 +0000 UTC m=+934.234035800"
Oct 30 13:18:44.074938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.053799    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.074938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.057456    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.177658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.143415    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.312666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.242343    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.346720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.247488    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:44.346720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.332097    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:44.346720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.332181    4291 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.513401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.513367    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:44.513697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.513672    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:44.684474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.684376    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:44.686821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.684701    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:18:44.687079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.686491    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:44.860662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.814069    4291 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded"
Oct 30 13:18:44.951102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.844826    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.951102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.935615    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 13:18:44.951102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.936130    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.951102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.936164    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:44.951102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.948748    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": context deadline exceeded"
Oct 30 13:18:44.951102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:44.950738    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:45.079938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.079893    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" probeResult="failure" output="Get \"https://10.133.0.42:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:45.080529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.080509    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:18:45.126732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.106783    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:45.126980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.126953    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:45.343496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.343472    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 13:18:45.438210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.433910    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 13:18:45.443173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.443145    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 13:18:45.446395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.446376    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 13:18:45.448586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.448570    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 13:18:45.536934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.536864    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:45.537178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.537148    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:45.549332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.549304    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 13:18:45.634716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.630883    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 13:18:45.726154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.726119    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:45.726409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.726382    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:18:45.734940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:45.734920    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 13:18:46.101794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:46.101332    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Oct 30 13:18:46.101794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:46.101396    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Oct 30 13:18:46.510202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:46.509716    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body=
Oct 30 13:18:46.510202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:46.509775    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused"
Oct 30 13:18:46.683187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:46.682746    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:46.683187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:46.682808    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:18:47.684496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:47.684057    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:47.684496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:47.684115    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:18:47.943027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:47.942943    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:47.943802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:47.943781    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:47.944866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:47.944846    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:18:48.044019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.043954    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:48.044395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.044369    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:48.045480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.045465    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:18:48.683659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.682520    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:48.683659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.682608    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:18:48.787519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.787481    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"]
Oct 30 13:18:48.803295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.803267    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"]
Oct 30 13:18:48.803477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.803404    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.817965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.817832    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:48.817965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.817872    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:48.817965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.817890    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:48.817965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.817923    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:18:48.834274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.834237    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.834459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.834284    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j847l\" (UniqueName: \"kubernetes.io/projected/305bb724-61d6-4d04-96f4-0349f3cd205c-kube-api-access-j847l\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.834459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.834428    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.935471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.935391    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.935471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.935438    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j847l\" (UniqueName: \"kubernetes.io/projected/305bb724-61d6-4d04-96f4-0349f3cd205c-kube-api-access-j847l\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.935734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.935521    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.936403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.936377    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.938130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.938105    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:48.977327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:48.977295    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j847l\" (UniqueName: \"kubernetes.io/projected/305bb724-61d6-4d04-96f4-0349f3cd205c-kube-api-access-j847l\") pod \"maas-api-revoke-sub-598e0cdd-wcxd4\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:49.119518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:49.119491    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:49.467659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:49.464790    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"]
Oct 30 13:18:49.478986 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:49.478949    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod305bb724_61d6_4d04_96f4_0349f3cd205c.slice/crio-b2fab9e22255e1420278d81d77be045da1ca68d8984b1badce023ccf3b4c49f8 WatchSource:0}: Error finding container b2fab9e22255e1420278d81d77be045da1ca68d8984b1badce023ccf3b4c49f8: Status 404 returned error can't find the container with id b2fab9e22255e1420278d81d77be045da1ca68d8984b1badce023ccf3b4c49f8
Oct 30 13:18:49.684537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:49.684057    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:49.684537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:49.684120    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:18:50.032122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.032081    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4" event={"ID":"305bb724-61d6-4d04-96f4-0349f3cd205c","Type":"ContainerStarted","Data":"e904d02fe2889a5a8747e1619f2b76767f2d29ac267e08b01d0d0155be8d9f02"}
Oct 30 13:18:50.032666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.032148    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4" event={"ID":"305bb724-61d6-4d04-96f4-0349f3cd205c","Type":"ContainerStarted","Data":"b2fab9e22255e1420278d81d77be045da1ca68d8984b1badce023ccf3b4c49f8"}
Oct 30 13:18:50.034563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.034522    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" event={"ID":"e47c1e82-2cff-4c8d-b702-05da93c3ba41","Type":"ContainerStarted","Data":"0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c"}
Oct 30 13:18:50.071653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.071540    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4" podStartSLOduration=2.07152501 podStartE2EDuration="2.07152501s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:50.070759227 +0000 UTC m=+940.330997497" watchObservedRunningTime="2026-10-30 13:18:50.07152501 +0000 UTC m=+940.331763243"
Oct 30 13:18:50.118105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.118046    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" podStartSLOduration=1.685383459 podStartE2EDuration="14.118027889s" totalImagesPullingTime="12.43264443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:50.11392758 +0000 UTC m=+940.374165815" watchObservedRunningTime="2026-10-30 13:18:50.118027889 +0000 UTC m=+940.378266136"
Oct 30 13:18:50.153841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.153795    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"]
Oct 30 13:18:50.157938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.157912    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.176571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.176394    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Oct 30 13:18:50.176571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.176445    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Oct 30 13:18:50.176571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.176451    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Oct 30 13:18:50.176571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.176525    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-fqpnj\""
Oct 30 13:18:50.178998 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.178971    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"]
Oct 30 13:18:50.253763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.253726    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.254017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.253820    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.254017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.253913    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.254147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.254046    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.254147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.254092    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.254147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.254117    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpsfg\" (UniqueName: \"kubernetes.io/projected/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-kube-api-access-vpsfg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.254308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.254278    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.357052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.357018    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.357270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.357085    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.357270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.357104    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.357270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.357168    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.357270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.357194    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.357270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.357217    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpsfg\" (UniqueName: \"kubernetes.io/projected/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-kube-api-access-vpsfg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.357270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.357271    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.359466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.359439    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.360404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.360369    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.361426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.361390    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.361620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.361596    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.362533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.362504    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.378197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.378169    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpsfg\" (UniqueName: \"kubernetes.io/projected/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-kube-api-access-vpsfg\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.378345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.378317    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-fbjhg\" (UID: \"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.470750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.470709    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:50.681441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.681357    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Oct 30 13:18:50.681441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.681422    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Oct 30 13:18:50.805385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:50.805353    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"]
Oct 30 13:18:50.805627 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:50.805606    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa8bb1b7_a64f_4b3a_9c05_638a5a08d60c.slice/crio-4fec49f793d6045a3af8b0854849f40ee7760a2b26bf1c34cb79fb7a66d68a04 WatchSource:0}: Error finding container 4fec49f793d6045a3af8b0854849f40ee7760a2b26bf1c34cb79fb7a66d68a04: Status 404 returned error can't find the container with id 4fec49f793d6045a3af8b0854849f40ee7760a2b26bf1c34cb79fb7a66d68a04
Oct 30 13:18:51.042489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:51.042377    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg" event={"ID":"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c","Type":"ContainerStarted","Data":"4fec49f793d6045a3af8b0854849f40ee7760a2b26bf1c34cb79fb7a66d68a04"}
Oct 30 13:18:51.044432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:51.044402    4291 generic.go:356] "Generic (PLEG): container finished" podID="305bb724-61d6-4d04-96f4-0349f3cd205c" containerID="e904d02fe2889a5a8747e1619f2b76767f2d29ac267e08b01d0d0155be8d9f02" exitCode=0
Oct 30 13:18:51.044669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:51.044447    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4" event={"ID":"305bb724-61d6-4d04-96f4-0349f3cd205c","Type":"ContainerDied","Data":"e904d02fe2889a5a8747e1619f2b76767f2d29ac267e08b01d0d0155be8d9f02"}
Oct 30 13:18:51.045019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:51.044994    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:18:51.685584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:51.685553    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:51.686012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:51.685993    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:51.686726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:51.686708    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 13:18:52.219417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.219386    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:52.293960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.293899    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/305bb724-61d6-4d04-96f4-0349f3cd205c-kube-api-access-j847l\" (UniqueName: \"kubernetes.io/projected/305bb724-61d6-4d04-96f4-0349f3cd205c-kube-api-access-j847l\") pod \"305bb724-61d6-4d04-96f4-0349f3cd205c\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") "
Oct 30 13:18:52.294217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.294192    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-xks-ca\") pod \"305bb724-61d6-4d04-96f4-0349f3cd205c\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") "
Oct 30 13:18:52.294332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.294261    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-openshift-ca\") pod \"305bb724-61d6-4d04-96f4-0349f3cd205c\" (UID: \"305bb724-61d6-4d04-96f4-0349f3cd205c\") "
Oct 30 13:18:52.295191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.295159    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-openshift-ca" pod "305bb724-61d6-4d04-96f4-0349f3cd205c" (UID: "305bb724-61d6-4d04-96f4-0349f3cd205c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:18:52.298252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.298221    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/305bb724-61d6-4d04-96f4-0349f3cd205c-kube-api-access-j847l" pod "305bb724-61d6-4d04-96f4-0349f3cd205c" (UID: "305bb724-61d6-4d04-96f4-0349f3cd205c"). InnerVolumeSpecName "kube-api-access-j847l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:18:52.315080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.315041    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-xks-ca" pod "305bb724-61d6-4d04-96f4-0349f3cd205c" (UID: "305bb724-61d6-4d04-96f4-0349f3cd205c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:18:52.395843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.395807    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:52.395843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.395841    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/305bb724-61d6-4d04-96f4-0349f3cd205c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:52.396108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:52.395856    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j847l\" (UniqueName: \"kubernetes.io/projected/305bb724-61d6-4d04-96f4-0349f3cd205c-kube-api-access-j847l\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:18:53.082095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:53.081947    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"
Oct 30 13:18:53.082095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:53.081963    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4" event={"ID":"305bb724-61d6-4d04-96f4-0349f3cd205c","Type":"ContainerDied","Data":"b2fab9e22255e1420278d81d77be045da1ca68d8984b1badce023ccf3b4c49f8"}
Oct 30 13:18:53.082095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:53.082017    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b2fab9e22255e1420278d81d77be045da1ca68d8984b1badce023ccf3b4c49f8"
Oct 30 13:18:54.550146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:54.550110    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:54.550475 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:54.550186    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:18:54.608345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:54.608313    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:18:55.100479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:55.100437    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg" event={"ID":"fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c","Type":"ContainerStarted","Data":"36e95ad967a76f007fa2810898ba2527221b7711f9f010a0b4e3d70741084e89"}
Oct 30 13:18:55.100726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:55.100670    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:55.102833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:55.102803    4291 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-fbjhg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:18:55.102946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:55.102861    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg" podUID="fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:18:55.155622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:55.155574    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg" podStartSLOduration=1.414746469 podStartE2EDuration="5.155560447s" totalImagesPullingTime="3.740813978s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:18:55.152718785 +0000 UTC m=+945.412957022" watchObservedRunningTime="2026-10-30 13:18:55.155560447 +0000 UTC m=+945.415798681"
Oct 30 13:18:56.125184 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:56.125149    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-fbjhg"
Oct 30 13:18:57.621660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:57.620753    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" probeResult="failure" output="Get \"https://10.133.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:18:59.228003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.210627    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"]
Oct 30 13:18:59.228820 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.228797    4291 cpu_manager.go:414] "containerMap: removing container" podUID="305bb724-61d6-4d04-96f4-0349f3cd205c" containerName="revoke-keys"
Oct 30 13:18:59.228820 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.228820    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="305bb724-61d6-4d04-96f4-0349f3cd205c" containerName="revoke-keys"
Oct 30 13:18:59.229009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.228975    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="305bb724-61d6-4d04-96f4-0349f3cd205c" containerName="revoke-keys"
Oct 30 13:18:59.238851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.236926    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.238851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.236919    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"]
Oct 30 13:18:59.272649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.272303    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:18:59.272649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.272363    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:18:59.272649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.272471    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:18:59.272649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.272522    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:18:59.346539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.346500    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d4af64-7a42-4ba0-a231-466918618781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.346750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.346563    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d4af64-7a42-4ba0-a231-466918618781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.346750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.346730    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpfbg\" (UniqueName: \"kubernetes.io/projected/52d4af64-7a42-4ba0-a231-466918618781-kube-api-access-hpfbg\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.447621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.447577    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d4af64-7a42-4ba0-a231-466918618781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.447841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.447696    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d4af64-7a42-4ba0-a231-466918618781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.447841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.447768    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpfbg\" (UniqueName: \"kubernetes.io/projected/52d4af64-7a42-4ba0-a231-466918618781-kube-api-access-hpfbg\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.448464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.448438    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d4af64-7a42-4ba0-a231-466918618781-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.450095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.450068    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d4af64-7a42-4ba0-a231-466918618781-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.467040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.467001    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpfbg\" (UniqueName: \"kubernetes.io/projected/52d4af64-7a42-4ba0-a231-466918618781-kube-api-access-hpfbg\") pod \"maas-api-revoke-sub-7d47bd17-tpngt\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.603790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.603751    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:18:59.911551 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:18:59.911516    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"]
Oct 30 13:18:59.912603 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:18:59.912576    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52d4af64_7a42_4ba0_a231_466918618781.slice/crio-e9b0f4708a8f4538c14add9c20d43aabb40a95a4ba4b1c2730eea089de911da3 WatchSource:0}: Error finding container e9b0f4708a8f4538c14add9c20d43aabb40a95a4ba4b1c2730eea089de911da3: Status 404 returned error can't find the container with id e9b0f4708a8f4538c14add9c20d43aabb40a95a4ba4b1c2730eea089de911da3
Oct 30 13:19:00.255135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:00.255099    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt" event={"ID":"52d4af64-7a42-4ba0-a231-466918618781","Type":"ContainerStarted","Data":"a71b0fab420657bcc40d0dbe7d0f7d6f35f1018e46d843621d85ed252221fd92"}
Oct 30 13:19:00.255553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:00.255165    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt" event={"ID":"52d4af64-7a42-4ba0-a231-466918618781","Type":"ContainerStarted","Data":"e9b0f4708a8f4538c14add9c20d43aabb40a95a4ba4b1c2730eea089de911da3"}
Oct 30 13:19:00.301266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:00.301189    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt" podStartSLOduration=1.3011728630000001 podStartE2EDuration="1.301172863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:18:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:00.294469634 +0000 UTC m=+950.554707869" watchObservedRunningTime="2026-10-30 13:19:00.301172863 +0000 UTC m=+950.561411151"
Oct 30 13:19:01.264309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:01.264277    4291 generic.go:356] "Generic (PLEG): container finished" podID="52d4af64-7a42-4ba0-a231-466918618781" containerID="a71b0fab420657bcc40d0dbe7d0f7d6f35f1018e46d843621d85ed252221fd92" exitCode=0
Oct 30 13:19:01.264741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:01.264362    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt" event={"ID":"52d4af64-7a42-4ba0-a231-466918618781","Type":"ContainerDied","Data":"a71b0fab420657bcc40d0dbe7d0f7d6f35f1018e46d843621d85ed252221fd92"}
Oct 30 13:19:02.055612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.055582    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:19:02.450946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.450916    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:19:02.487742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.487199    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52d4af64-7a42-4ba0-a231-466918618781-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d4af64-7a42-4ba0-a231-466918618781-maas-api-openshift-ca\") pod \"52d4af64-7a42-4ba0-a231-466918618781\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") "
Oct 30 13:19:02.487742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.487404    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52d4af64-7a42-4ba0-a231-466918618781-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d4af64-7a42-4ba0-a231-466918618781-maas-api-xks-ca\") pod \"52d4af64-7a42-4ba0-a231-466918618781\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") "
Oct 30 13:19:02.487742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.487440    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52d4af64-7a42-4ba0-a231-466918618781-kube-api-access-hpfbg\" (UniqueName: \"kubernetes.io/projected/52d4af64-7a42-4ba0-a231-466918618781-kube-api-access-hpfbg\") pod \"52d4af64-7a42-4ba0-a231-466918618781\" (UID: \"52d4af64-7a42-4ba0-a231-466918618781\") "
Oct 30 13:19:02.487742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.487603    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52d4af64-7a42-4ba0-a231-466918618781-maas-api-openshift-ca" pod "52d4af64-7a42-4ba0-a231-466918618781" (UID: "52d4af64-7a42-4ba0-a231-466918618781"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:02.488237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.488211    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52d4af64-7a42-4ba0-a231-466918618781-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:02.490349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.490318    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52d4af64-7a42-4ba0-a231-466918618781-maas-api-xks-ca" pod "52d4af64-7a42-4ba0-a231-466918618781" (UID: "52d4af64-7a42-4ba0-a231-466918618781"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:02.490807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.490768    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52d4af64-7a42-4ba0-a231-466918618781-kube-api-access-hpfbg" pod "52d4af64-7a42-4ba0-a231-466918618781" (UID: "52d4af64-7a42-4ba0-a231-466918618781"). InnerVolumeSpecName "kube-api-access-hpfbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:02.589620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.589528    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52d4af64-7a42-4ba0-a231-466918618781-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:02.589620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:02.589559    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpfbg\" (UniqueName: \"kubernetes.io/projected/52d4af64-7a42-4ba0-a231-466918618781-kube-api-access-hpfbg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:03.282259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:03.282182    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"
Oct 30 13:19:03.282259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:03.282214    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt" event={"ID":"52d4af64-7a42-4ba0-a231-466918618781","Type":"ContainerDied","Data":"e9b0f4708a8f4538c14add9c20d43aabb40a95a4ba4b1c2730eea089de911da3"}
Oct 30 13:19:03.282438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:03.282262    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e9b0f4708a8f4538c14add9c20d43aabb40a95a4ba4b1c2730eea089de911da3"
Oct 30 13:19:08.211938 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:19:08.211886    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:19:10.133709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:10.133680    4291 scope.go:122] "RemoveContainer" containerID="a8c1f06c7828835b08589c9012356eeb47e393970b12a9732dd0b7fe46b47dd0"
Oct 30 13:19:10.143707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:10.143687    4291 scope.go:122] "RemoveContainer" containerID="7f1ec032fe417d8027ea9b3b4d2f8e71bf68373853fc9f1c200f13944cb93fb1"
Oct 30 13:19:10.154174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:10.154153    4291 scope.go:122] "RemoveContainer" containerID="7d1f5599ed55cd4d5c1766e55386b9ffe2d0ba9c8271db86910e75c6215492a9"
Oct 30 13:19:10.163149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:10.163127    4291 scope.go:122] "RemoveContainer" containerID="69b0ed4570ce76109a824a78f2d8383c7925fdddd544541e70e6c201b1be36c7"
Oct 30 13:19:10.171801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:10.171782    4291 scope.go:122] "RemoveContainer" containerID="6a54819ffd60b9e618dba0114fd2e49b8bb9a6574ee1adbe6f684549e4ad2fae"
Oct 30 13:19:13.486870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.486833    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"]
Oct 30 13:19:13.487396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.487372    4291 cpu_manager.go:414] "containerMap: removing container" podUID="52d4af64-7a42-4ba0-a231-466918618781" containerName="revoke-keys"
Oct 30 13:19:13.487396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.487385    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52d4af64-7a42-4ba0-a231-466918618781" containerName="revoke-keys"
Oct 30 13:19:13.487509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.487494    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="52d4af64-7a42-4ba0-a231-466918618781" containerName="revoke-keys"
Oct 30 13:19:13.492789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.492768    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.508816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.508785    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"]
Oct 30 13:19:13.514572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.514539    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:19:13.515398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.515374    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:19:13.515533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.515462    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:19:13.515533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.515527    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:19:13.649080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.649030    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9hrc\" (UniqueName: \"kubernetes.io/projected/6dec60b5-b084-4df0-84dd-8bc949a77d32-kube-api-access-v9hrc\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.649305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.649288    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.649426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.649406    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.750796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.750691    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.750972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.750851    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9hrc\" (UniqueName: \"kubernetes.io/projected/6dec60b5-b084-4df0-84dd-8bc949a77d32-kube-api-access-v9hrc\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.751042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.750990    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.751514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.751476    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.753668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.753629    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.770724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.770686    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9hrc\" (UniqueName: \"kubernetes.io/projected/6dec60b5-b084-4df0-84dd-8bc949a77d32-kube-api-access-v9hrc\") pod \"maas-api-revoke-sub-4d72275a-2lpr9\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:13.808872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:13.808830    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:14.137699 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:19:14.135851    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dec60b5_b084_4df0_84dd_8bc949a77d32.slice/crio-37c8834890a0cccf006187e608652239e10d68dd006a9adf86a7d427064e1e74 WatchSource:0}: Error finding container 37c8834890a0cccf006187e608652239e10d68dd006a9adf86a7d427064e1e74: Status 404 returned error can't find the container with id 37c8834890a0cccf006187e608652239e10d68dd006a9adf86a7d427064e1e74
Oct 30 13:19:14.143891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:14.143865    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"]
Oct 30 13:19:14.379986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:14.379945    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9" event={"ID":"6dec60b5-b084-4df0-84dd-8bc949a77d32","Type":"ContainerStarted","Data":"37c8834890a0cccf006187e608652239e10d68dd006a9adf86a7d427064e1e74"}
Oct 30 13:19:15.397585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:15.397550    4291 generic.go:356] "Generic (PLEG): container finished" podID="6dec60b5-b084-4df0-84dd-8bc949a77d32" containerID="e099c2ee51aab1e4410e0cff3443962ecadf4933ff5ee381f346f16c0cfc56fe" exitCode=0
Oct 30 13:19:15.398138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:15.397618    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9" event={"ID":"6dec60b5-b084-4df0-84dd-8bc949a77d32","Type":"ContainerDied","Data":"e099c2ee51aab1e4410e0cff3443962ecadf4933ff5ee381f346f16c0cfc56fe"}
Oct 30 13:19:16.607109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.607081    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:16.720524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.720408    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6dec60b5-b084-4df0-84dd-8bc949a77d32-kube-api-access-v9hrc\" (UniqueName: \"kubernetes.io/projected/6dec60b5-b084-4df0-84dd-8bc949a77d32-kube-api-access-v9hrc\") pod \"6dec60b5-b084-4df0-84dd-8bc949a77d32\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") "
Oct 30 13:19:16.720724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.720564    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-openshift-ca\") pod \"6dec60b5-b084-4df0-84dd-8bc949a77d32\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") "
Oct 30 13:19:16.720802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.720752    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-xks-ca\") pod \"6dec60b5-b084-4df0-84dd-8bc949a77d32\" (UID: \"6dec60b5-b084-4df0-84dd-8bc949a77d32\") "
Oct 30 13:19:16.721001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.720972    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-openshift-ca" pod "6dec60b5-b084-4df0-84dd-8bc949a77d32" (UID: "6dec60b5-b084-4df0-84dd-8bc949a77d32"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:16.721287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.721268    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:16.722622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.722599    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dec60b5-b084-4df0-84dd-8bc949a77d32-kube-api-access-v9hrc" pod "6dec60b5-b084-4df0-84dd-8bc949a77d32" (UID: "6dec60b5-b084-4df0-84dd-8bc949a77d32"). InnerVolumeSpecName "kube-api-access-v9hrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:16.722826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.722803    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-xks-ca" pod "6dec60b5-b084-4df0-84dd-8bc949a77d32" (UID: "6dec60b5-b084-4df0-84dd-8bc949a77d32"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:16.823138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.823100    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9hrc\" (UniqueName: \"kubernetes.io/projected/6dec60b5-b084-4df0-84dd-8bc949a77d32-kube-api-access-v9hrc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:16.823138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:16.823132    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6dec60b5-b084-4df0-84dd-8bc949a77d32-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:17.416770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:17.416725    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9" event={"ID":"6dec60b5-b084-4df0-84dd-8bc949a77d32","Type":"ContainerDied","Data":"37c8834890a0cccf006187e608652239e10d68dd006a9adf86a7d427064e1e74"}
Oct 30 13:19:17.416919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:17.416795    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37c8834890a0cccf006187e608652239e10d68dd006a9adf86a7d427064e1e74"
Oct 30 13:19:17.416919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:17.416744    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"
Oct 30 13:19:21.098980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:21.098942    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"]
Oct 30 13:19:21.110895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:21.110853    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2a0c5b8-x4j87"]
Oct 30 13:19:21.879622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:21.879584    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f19001ae-04a7-431a-8d1c-06ec4d981a71" path="/var/lib/kubelet/pods/f19001ae-04a7-431a-8d1c-06ec4d981a71/volumes"
Oct 30 13:19:23.903562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.903531    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"]
Oct 30 13:19:23.904088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.904074    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6dec60b5-b084-4df0-84dd-8bc949a77d32" containerName="revoke-keys"
Oct 30 13:19:23.904132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.904088    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6dec60b5-b084-4df0-84dd-8bc949a77d32" containerName="revoke-keys"
Oct 30 13:19:23.904181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.904173    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6dec60b5-b084-4df0-84dd-8bc949a77d32" containerName="revoke-keys"
Oct 30 13:19:23.912143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.912124    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:23.924513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.924483    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"]
Oct 30 13:19:23.931180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.931155    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:19:23.931450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.931432    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:19:23.931570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.931490    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:19:23.937825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:23.937803    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:19:24.033953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.033914    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.034130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.033969    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.034130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.034063    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr9sn\" (UniqueName: \"kubernetes.io/projected/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-kube-api-access-qr9sn\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.135013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.134971    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.135228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.135033    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.135228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.135076    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr9sn\" (UniqueName: \"kubernetes.io/projected/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-kube-api-access-qr9sn\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.135682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.135661    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.137396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.137378    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.154552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.154479    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr9sn\" (UniqueName: \"kubernetes.io/projected/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-kube-api-access-qr9sn\") pod \"maas-api-revoke-sub-cd613a5e-8rf66\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.227940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.227905    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:24.539349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:24.539312    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"]
Oct 30 13:19:25.478777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:25.478749    4291 generic.go:356] "Generic (PLEG): container finished" podID="19a69509-7ff0-4be0-a1c1-8ce7e07573c2" containerID="7eaef2a957cd05b8723a306590a789cc852c2c70176eb6c20d8e2c664e1980b4" exitCode=0
Oct 30 13:19:25.479102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:25.478800    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66" event={"ID":"19a69509-7ff0-4be0-a1c1-8ce7e07573c2","Type":"ContainerDied","Data":"7eaef2a957cd05b8723a306590a789cc852c2c70176eb6c20d8e2c664e1980b4"}
Oct 30 13:19:25.479102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:25.478835    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66" event={"ID":"19a69509-7ff0-4be0-a1c1-8ce7e07573c2","Type":"ContainerStarted","Data":"692386ca1ca6224b810a7fd6d50c04f45e80011fed458b7c4c7e3207cc773b81"}
Oct 30 13:19:26.647386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.647359    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:26.770111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.770039    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-kube-api-access-qr9sn\" (UniqueName: \"kubernetes.io/projected/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-kube-api-access-qr9sn\") pod \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") "
Oct 30 13:19:26.770296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.770273    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-openshift-ca\") pod \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") "
Oct 30 13:19:26.770403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.770385    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-xks-ca\") pod \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\" (UID: \"19a69509-7ff0-4be0-a1c1-8ce7e07573c2\") "
Oct 30 13:19:26.770687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.770628    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-openshift-ca" pod "19a69509-7ff0-4be0-a1c1-8ce7e07573c2" (UID: "19a69509-7ff0-4be0-a1c1-8ce7e07573c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:26.770952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.770935    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:26.772179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.772155    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-kube-api-access-qr9sn" pod "19a69509-7ff0-4be0-a1c1-8ce7e07573c2" (UID: "19a69509-7ff0-4be0-a1c1-8ce7e07573c2"). InnerVolumeSpecName "kube-api-access-qr9sn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:26.772392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.772368    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-xks-ca" pod "19a69509-7ff0-4be0-a1c1-8ce7e07573c2" (UID: "19a69509-7ff0-4be0-a1c1-8ce7e07573c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:26.872498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.872462    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:26.872666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:26.872508    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr9sn\" (UniqueName: \"kubernetes.io/projected/19a69509-7ff0-4be0-a1c1-8ce7e07573c2-kube-api-access-qr9sn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:27.512089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:27.512063    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"
Oct 30 13:19:27.512258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:27.512106    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66" event={"ID":"19a69509-7ff0-4be0-a1c1-8ce7e07573c2","Type":"ContainerDied","Data":"692386ca1ca6224b810a7fd6d50c04f45e80011fed458b7c4c7e3207cc773b81"}
Oct 30 13:19:27.512258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:27.512164    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="692386ca1ca6224b810a7fd6d50c04f45e80011fed458b7c4c7e3207cc773b81"
Oct 30 13:19:36.310350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:36.310316    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:19:37.037872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:37.037824    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"]
Oct 30 13:19:37.058663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:37.057661    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cccafb7-vvk5x"]
Oct 30 13:19:37.908660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:37.907612    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43901a56-9404-4f14-9962-9540261123f9" path="/var/lib/kubelet/pods/43901a56-9404-4f14-9962-9540261123f9/volumes"
Oct 30 13:19:38.247839 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:19:38.247751    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:19:45.035883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:45.035849    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"]
Oct 30 13:19:45.047417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:45.047387    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f3731ddd-5lpp6"]
Oct 30 13:19:45.879345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:45.879306    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f4c9845-91fe-49ec-be3d-6f04d0fbc864" path="/var/lib/kubelet/pods/7f4c9845-91fe-49ec-be3d-6f04d0fbc864/volumes"
Oct 30 13:19:51.033590 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:51.033554    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"]
Oct 30 13:19:51.046012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:51.045980    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7bac55-vjp74"]
Oct 30 13:19:51.880860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:51.880823    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="504c5f3c-8117-4a37-9de3-a198f3f48812" path="/var/lib/kubelet/pods/504c5f3c-8117-4a37-9de3-a198f3f48812/volumes"
Oct 30 13:19:53.301135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.301095    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"]
Oct 30 13:19:53.301670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.301648    4291 cpu_manager.go:414] "containerMap: removing container" podUID="19a69509-7ff0-4be0-a1c1-8ce7e07573c2" containerName="revoke-keys"
Oct 30 13:19:53.301670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.301664    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19a69509-7ff0-4be0-a1c1-8ce7e07573c2" containerName="revoke-keys"
Oct 30 13:19:53.301789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.301780    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="19a69509-7ff0-4be0-a1c1-8ce7e07573c2" containerName="revoke-keys"
Oct 30 13:19:53.306755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.306730    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.322307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.322273    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"]
Oct 30 13:19:53.329659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.328626    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:19:53.329659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.328676    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:19:53.329659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.328709    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:19:53.334496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.334463    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:19:53.474767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.474722    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.475031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.475005    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.475094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.475075    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tffql\" (UniqueName: \"kubernetes.io/projected/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-kube-api-access-tffql\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.576095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.575993    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.576095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.576068    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tffql\" (UniqueName: \"kubernetes.io/projected/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-kube-api-access-tffql\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.576349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.576100    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.576754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.576729    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.578468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.578448    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.595058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.595022    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tffql\" (UniqueName: \"kubernetes.io/projected/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-kube-api-access-tffql\") pod \"maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.627144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.627108    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:53.953865 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:19:53.953818    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e91cca7_24b3_426a_b5dd_f69e8fc4cc18.slice/crio-1463d6a0008e8cce6e51fd0d69539a78cfb1e4b9de4b28d6113a8b03a49d7c46 WatchSource:0}: Error finding container 1463d6a0008e8cce6e51fd0d69539a78cfb1e4b9de4b28d6113a8b03a49d7c46: Status 404 returned error can't find the container with id 1463d6a0008e8cce6e51fd0d69539a78cfb1e4b9de4b28d6113a8b03a49d7c46
Oct 30 13:19:53.955736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:53.955705    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"]
Oct 30 13:19:54.750401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:54.750358    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k" event={"ID":"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18","Type":"ContainerStarted","Data":"04687bb05526c06a912ce34ff70a2fd443ffe0063ea6d83f73bdda40389006a1"}
Oct 30 13:19:54.750935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:54.750426    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k" event={"ID":"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18","Type":"ContainerStarted","Data":"1463d6a0008e8cce6e51fd0d69539a78cfb1e4b9de4b28d6113a8b03a49d7c46"}
Oct 30 13:19:54.790905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:54.790839    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k" podStartSLOduration=1.790819983 podStartE2EDuration="1.790819983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:19:54.78729064 +0000 UTC m=+1005.047528875" watchObservedRunningTime="2026-10-30 13:19:54.790819983 +0000 UTC m=+1005.051058211"
Oct 30 13:19:54.927299 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:19:54.927258    4291 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e91cca7_24b3_426a_b5dd_f69e8fc4cc18.slice/crio-04687bb05526c06a912ce34ff70a2fd443ffe0063ea6d83f73bdda40389006a1.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e91cca7_24b3_426a_b5dd_f69e8fc4cc18.slice/crio-conmon-04687bb05526c06a912ce34ff70a2fd443ffe0063ea6d83f73bdda40389006a1.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:19:55.763908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:55.763867    4291 generic.go:356] "Generic (PLEG): container finished" podID="2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" containerID="04687bb05526c06a912ce34ff70a2fd443ffe0063ea6d83f73bdda40389006a1" exitCode=0
Oct 30 13:19:55.764343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:55.763949    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k" event={"ID":"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18","Type":"ContainerDied","Data":"04687bb05526c06a912ce34ff70a2fd443ffe0063ea6d83f73bdda40389006a1"}
Oct 30 13:19:57.198918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.198871    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:19:57.401721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.401355    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-xks-ca\") pod \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") "
Oct 30 13:19:57.401721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.401440    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-kube-api-access-tffql\" (UniqueName: \"kubernetes.io/projected/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-kube-api-access-tffql\") pod \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") "
Oct 30 13:19:57.401721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.401702    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-openshift-ca\") pod \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\" (UID: \"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18\") "
Oct 30 13:19:57.424859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.402579    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-openshift-ca" pod "2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" (UID: "2e91cca7-24b3-426a-b5dd-f69e8fc4cc18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:19:57.426031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.425998    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-xks-ca" pod "2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" (UID: "2e91cca7-24b3-426a-b5dd-f69e8fc4cc18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:19:57.426268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.426246    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-kube-api-access-tffql" pod "2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" (UID: "2e91cca7-24b3-426a-b5dd-f69e8fc4cc18"). InnerVolumeSpecName "kube-api-access-tffql". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:19:57.515671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.515090    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:57.515671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.515129    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:57.515671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.515147    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tffql\" (UniqueName: \"kubernetes.io/projected/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18-kube-api-access-tffql\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:19:57.810718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.805143    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k" event={"ID":"2e91cca7-24b3-426a-b5dd-f69e8fc4cc18","Type":"ContainerDied","Data":"1463d6a0008e8cce6e51fd0d69539a78cfb1e4b9de4b28d6113a8b03a49d7c46"}
Oct 30 13:19:57.810718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.805224    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1463d6a0008e8cce6e51fd0d69539a78cfb1e4b9de4b28d6113a8b03a49d7c46"
Oct 30 13:19:57.810718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:19:57.805296    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"
Oct 30 13:20:00.035360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:00.035323    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"]
Oct 30 13:20:00.047180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:00.047152    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23ef410d-mz2nn"]
Oct 30 13:20:01.879463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:01.879427    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f725e253-a9ce-43ab-88ed-3a373c32af71" path="/var/lib/kubelet/pods/f725e253-a9ce-43ab-88ed-3a373c32af71/volumes"
Oct 30 13:20:02.033701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:02.033668    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"]
Oct 30 13:20:02.045355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:02.045314    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eae4bc3f-k772s"]
Oct 30 13:20:03.883300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:03.883260    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="972f6718-00ca-47e1-9ccb-c52d884ed69f" path="/var/lib/kubelet/pods/972f6718-00ca-47e1-9ccb-c52d884ed69f/volumes"
Oct 30 13:20:06.245746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.245711    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"]
Oct 30 13:20:06.246245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.246231    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" containerName="revoke-keys"
Oct 30 13:20:06.246289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.246246    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" containerName="revoke-keys"
Oct 30 13:20:06.246381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.246372    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" containerName="revoke-keys"
Oct 30 13:20:06.250054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.250032    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.265766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.265739    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"]
Oct 30 13:20:06.274661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.272585    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:06.274661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.272599    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:20:06.274661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.272600    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:20:06.275963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.275941    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:20:06.363126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.363090    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.363340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.363159    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9wt\" (UniqueName: \"kubernetes.io/projected/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-kube-api-access-6v9wt\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.363340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.363327    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.464313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.464274    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.464488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.464359    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9wt\" (UniqueName: \"kubernetes.io/projected/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-kube-api-access-6v9wt\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.464542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.464517    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.464955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.464936    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.466908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.466889    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.483450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.483428    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9wt\" (UniqueName: \"kubernetes.io/projected/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-kube-api-access-6v9wt\") pod \"maas-api-revoke-sub-10a94e4f-bjb7m\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.564345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.564261    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:06.871787 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:06.871760    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"]
Oct 30 13:20:06.872251 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:20:06.872227    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb61b6b8_22be_4332_bd9d_3bd93997aaf5.slice/crio-6d575123ad70768128666a0ee132f7fb2cff3efa6e2f2aeb35741e394d63d490 WatchSource:0}: Error finding container 6d575123ad70768128666a0ee132f7fb2cff3efa6e2f2aeb35741e394d63d490: Status 404 returned error can't find the container with id 6d575123ad70768128666a0ee132f7fb2cff3efa6e2f2aeb35741e394d63d490
Oct 30 13:20:07.899887 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:07.899850    4291 generic.go:356] "Generic (PLEG): container finished" podID="bb61b6b8-22be-4332-bd9d-3bd93997aaf5" containerID="bce45a69d907e6b2694353c4df369045a6cc9b223eb31dee683583d1b27e136b" exitCode=0
Oct 30 13:20:07.900294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:07.899935    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m" event={"ID":"bb61b6b8-22be-4332-bd9d-3bd93997aaf5","Type":"ContainerDied","Data":"bce45a69d907e6b2694353c4df369045a6cc9b223eb31dee683583d1b27e136b"}
Oct 30 13:20:07.900294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:07.899988    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m" event={"ID":"bb61b6b8-22be-4332-bd9d-3bd93997aaf5","Type":"ContainerStarted","Data":"6d575123ad70768128666a0ee132f7fb2cff3efa6e2f2aeb35741e394d63d490"}
Oct 30 13:20:08.210525 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:20:08.210488    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:20:09.033388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.033316    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"]
Oct 30 13:20:09.044993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.044967    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c90ecd6f-86xst"]
Oct 30 13:20:09.057057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.057037    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:09.199224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.199135    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-openshift-ca\") pod \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") "
Oct 30 13:20:09.199397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.199249    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-xks-ca\") pod \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") "
Oct 30 13:20:09.199397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.199300    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-kube-api-access-6v9wt\" (UniqueName: \"kubernetes.io/projected/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-kube-api-access-6v9wt\") pod \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\" (UID: \"bb61b6b8-22be-4332-bd9d-3bd93997aaf5\") "
Oct 30 13:20:09.199566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.199538    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-openshift-ca" pod "bb61b6b8-22be-4332-bd9d-3bd93997aaf5" (UID: "bb61b6b8-22be-4332-bd9d-3bd93997aaf5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:09.199906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.199879    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:09.201799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.201768    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-xks-ca" pod "bb61b6b8-22be-4332-bd9d-3bd93997aaf5" (UID: "bb61b6b8-22be-4332-bd9d-3bd93997aaf5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:09.201933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.201912    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-kube-api-access-6v9wt" pod "bb61b6b8-22be-4332-bd9d-3bd93997aaf5" (UID: "bb61b6b8-22be-4332-bd9d-3bd93997aaf5"). InnerVolumeSpecName "kube-api-access-6v9wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:09.300793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.300755    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:09.300793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.300789    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9wt\" (UniqueName: \"kubernetes.io/projected/bb61b6b8-22be-4332-bd9d-3bd93997aaf5-kube-api-access-6v9wt\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:09.879346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.879311    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="570c8147-20e0-46df-905b-b5d43038cd74" path="/var/lib/kubelet/pods/570c8147-20e0-46df-905b-b5d43038cd74/volumes"
Oct 30 13:20:09.915266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.915240    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"
Oct 30 13:20:09.915435 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.915240    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m" event={"ID":"bb61b6b8-22be-4332-bd9d-3bd93997aaf5","Type":"ContainerDied","Data":"6d575123ad70768128666a0ee132f7fb2cff3efa6e2f2aeb35741e394d63d490"}
Oct 30 13:20:09.915435 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:09.915358    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d575123ad70768128666a0ee132f7fb2cff3efa6e2f2aeb35741e394d63d490"
Oct 30 13:20:14.668664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:14.668616    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:20:19.033418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:19.033382    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"]
Oct 30 13:20:19.044419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:19.044392    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3bb9da49-xhlzd"]
Oct 30 13:20:19.880255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:19.880220    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9ee5308a-adbc-49d3-89f4-46cc2cc4f400" path="/var/lib/kubelet/pods/9ee5308a-adbc-49d3-89f4-46cc2cc4f400/volumes"
Oct 30 13:20:25.786273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.786188    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"]
Oct 30 13:20:25.786795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.786778    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bb61b6b8-22be-4332-bd9d-3bd93997aaf5" containerName="revoke-keys"
Oct 30 13:20:25.786861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.786796    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb61b6b8-22be-4332-bd9d-3bd93997aaf5" containerName="revoke-keys"
Oct 30 13:20:25.786906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.786897    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb61b6b8-22be-4332-bd9d-3bd93997aaf5" containerName="revoke-keys"
Oct 30 13:20:25.791609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.791593    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:25.806439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.806415    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"]
Oct 30 13:20:25.813585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.813559    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:20:25.813585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.813582    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:25.813813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.813587    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:20:25.813813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.813559    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:20:25.933314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.933280    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dxp\" (UniqueName: \"kubernetes.io/projected/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-kube-api-access-b2dxp\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:25.933495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.933433    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:25.933541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:25.933518    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.034835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.034776    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.035100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.034921    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dxp\" (UniqueName: \"kubernetes.io/projected/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-kube-api-access-b2dxp\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.035100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.035051    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.035608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.035577    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.037954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.037896    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.053622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.053590    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dxp\" (UniqueName: \"kubernetes.io/projected/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-kube-api-access-b2dxp\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.106699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.106669    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:26.418080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:26.418026    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"]
Oct 30 13:20:27.044391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:27.044287    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7" event={"ID":"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e","Type":"ContainerStarted","Data":"11dfa3ad037128d02a281a15553ebba28c61706d598b275adb7ee5968419df45"}
Oct 30 13:20:27.044391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:27.044357    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7" event={"ID":"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e","Type":"ContainerStarted","Data":"ed9f448efd0298787942485f45f69377c034093b1b19c13952ac62be7c0b1437"}
Oct 30 13:20:27.088837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:27.088781    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7" podStartSLOduration=2.08876567 podStartE2EDuration="2.08876567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:27.084067286 +0000 UTC m=+1037.344305520" watchObservedRunningTime="2026-10-30 13:20:27.08876567 +0000 UTC m=+1037.349003903"
Oct 30 13:20:27.253505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:27.253468    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"]
Oct 30 13:20:27.257591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:27.257567    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9fdad5-4zn9k"]
Oct 30 13:20:27.879200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:27.879161    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e91cca7-24b3-426a-b5dd-f69e8fc4cc18" path="/var/lib/kubelet/pods/2e91cca7-24b3-426a-b5dd-f69e8fc4cc18/volumes"
Oct 30 13:20:28.053185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:28.053152    4291 generic.go:356] "Generic (PLEG): container finished" podID="aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" containerID="11dfa3ad037128d02a281a15553ebba28c61706d598b275adb7ee5968419df45" exitCode=0
Oct 30 13:20:28.053703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:28.053238    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7" event={"ID":"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e","Type":"ContainerDied","Data":"11dfa3ad037128d02a281a15553ebba28c61706d598b275adb7ee5968419df45"}
Oct 30 13:20:29.236876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.236849    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:29.377449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.377420    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-openshift-ca\") pod \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") "
Oct 30 13:20:29.377649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.377507    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-xks-ca\") pod \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") "
Oct 30 13:20:29.377703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.377681    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-kube-api-access-b2dxp\" (UniqueName: \"kubernetes.io/projected/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-kube-api-access-b2dxp\") pod \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\" (UID: \"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e\") "
Oct 30 13:20:29.377832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.377808    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-openshift-ca" pod "aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" (UID: "aa9099ad-07db-440c-b4b6-f0e1d71e4e9e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:29.378120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.378105    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:29.379596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.379573    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-xks-ca" pod "aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" (UID: "aa9099ad-07db-440c-b4b6-f0e1d71e4e9e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:29.379802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.379778    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-kube-api-access-b2dxp" pod "aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" (UID: "aa9099ad-07db-440c-b4b6-f0e1d71e4e9e"). InnerVolumeSpecName "kube-api-access-b2dxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:29.479525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.479451    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2dxp\" (UniqueName: \"kubernetes.io/projected/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-kube-api-access-b2dxp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:29.479525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:29.479503    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:30.069033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:30.068998    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7" event={"ID":"aa9099ad-07db-440c-b4b6-f0e1d71e4e9e","Type":"ContainerDied","Data":"ed9f448efd0298787942485f45f69377c034093b1b19c13952ac62be7c0b1437"}
Oct 30 13:20:30.069198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:30.069053    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9f448efd0298787942485f45f69377c034093b1b19c13952ac62be7c0b1437"
Oct 30 13:20:30.069198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:30.069009    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"
Oct 30 13:20:35.388757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.388718    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"]
Oct 30 13:20:35.389314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.389298    4291 cpu_manager.go:414] "containerMap: removing container" podUID="aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" containerName="revoke-keys"
Oct 30 13:20:35.389360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.389315    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" containerName="revoke-keys"
Oct 30 13:20:35.389431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.389421    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" containerName="revoke-keys"
Oct 30 13:20:35.393239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.393218    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.411005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.410974    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"]
Oct 30 13:20:35.418233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.418204    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:35.418411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.418296    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:20:35.418411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.418334    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:20:35.419718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.419694    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:20:35.565939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.565904    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.566130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.566021    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.566130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.566075    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbvtd\" (UniqueName: \"kubernetes.io/projected/4d50d47e-e814-460f-a358-5fe461a0bd90-kube-api-access-mbvtd\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.667905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.667813    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.668059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.667924    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.668059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.667993    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbvtd\" (UniqueName: \"kubernetes.io/projected/4d50d47e-e814-460f-a358-5fe461a0bd90-kube-api-access-mbvtd\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.668543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.668513    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.670388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.670367    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.688143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.688092    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbvtd\" (UniqueName: \"kubernetes.io/projected/4d50d47e-e814-460f-a358-5fe461a0bd90-kube-api-access-mbvtd\") pod \"maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:35.709020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:35.708989    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:36.018125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:36.018091    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"]
Oct 30 13:20:36.018569 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:20:36.018543    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d50d47e_e814_460f_a358_5fe461a0bd90.slice/crio-f5462f73a4735b69687f61e112aca02759a5232de4c54c0febe46d87da60f873 WatchSource:0}: Error finding container f5462f73a4735b69687f61e112aca02759a5232de4c54c0febe46d87da60f873: Status 404 returned error can't find the container with id f5462f73a4735b69687f61e112aca02759a5232de4c54c0febe46d87da60f873
Oct 30 13:20:36.114615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:36.114572    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk" event={"ID":"4d50d47e-e814-460f-a358-5fe461a0bd90","Type":"ContainerStarted","Data":"f5462f73a4735b69687f61e112aca02759a5232de4c54c0febe46d87da60f873"}
Oct 30 13:20:36.269569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:36.269538    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"]
Oct 30 13:20:37.250401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:37.249740    4291 generic.go:356] "Generic (PLEG): container finished" podID="4d50d47e-e814-460f-a358-5fe461a0bd90" containerID="0a8b563dc72ae429e854dbc9a9e3a11616bbfaa1af71d7aa7764beb9568747b4" exitCode=0
Oct 30 13:20:37.250401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:37.250022    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk" event={"ID":"4d50d47e-e814-460f-a358-5fe461a0bd90","Type":"ContainerDied","Data":"0a8b563dc72ae429e854dbc9a9e3a11616bbfaa1af71d7aa7764beb9568747b4"}
Oct 30 13:20:38.540733 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:20:38.540624    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:20:38.871454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:38.871420    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:38.981559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:38.981453    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d50d47e-e814-460f-a358-5fe461a0bd90-kube-api-access-mbvtd\" (UniqueName: \"kubernetes.io/projected/4d50d47e-e814-460f-a358-5fe461a0bd90-kube-api-access-mbvtd\") pod \"4d50d47e-e814-460f-a358-5fe461a0bd90\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") "
Oct 30 13:20:38.981559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:38.981515    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-xks-ca\") pod \"4d50d47e-e814-460f-a358-5fe461a0bd90\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") "
Oct 30 13:20:38.981835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:38.981779    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-openshift-ca\") pod \"4d50d47e-e814-460f-a358-5fe461a0bd90\" (UID: \"4d50d47e-e814-460f-a358-5fe461a0bd90\") "
Oct 30 13:20:38.982896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:38.982847    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-openshift-ca" pod "4d50d47e-e814-460f-a358-5fe461a0bd90" (UID: "4d50d47e-e814-460f-a358-5fe461a0bd90"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:38.985589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:38.985553    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d50d47e-e814-460f-a358-5fe461a0bd90-kube-api-access-mbvtd" pod "4d50d47e-e814-460f-a358-5fe461a0bd90" (UID: "4d50d47e-e814-460f-a358-5fe461a0bd90"). InnerVolumeSpecName "kube-api-access-mbvtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:38.985743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:38.985596    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-xks-ca" pod "4d50d47e-e814-460f-a358-5fe461a0bd90" (UID: "4d50d47e-e814-460f-a358-5fe461a0bd90"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:39.083609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:39.083572    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:39.083609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:39.083603    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbvtd\" (UniqueName: \"kubernetes.io/projected/4d50d47e-e814-460f-a358-5fe461a0bd90-kube-api-access-mbvtd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:39.083609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:39.083613    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d50d47e-e814-460f-a358-5fe461a0bd90-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:39.423170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:39.423139    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"
Oct 30 13:20:39.423366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:39.423170    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk" event={"ID":"4d50d47e-e814-460f-a358-5fe461a0bd90","Type":"ContainerDied","Data":"f5462f73a4735b69687f61e112aca02759a5232de4c54c0febe46d87da60f873"}
Oct 30 13:20:39.423366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:39.423251    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5462f73a4735b69687f61e112aca02759a5232de4c54c0febe46d87da60f873"
Oct 30 13:20:41.145806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.145771    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"]
Oct 30 13:20:41.146324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.146309    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4d50d47e-e814-460f-a358-5fe461a0bd90" containerName="revoke-keys"
Oct 30 13:20:41.146385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.146325    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d50d47e-e814-460f-a358-5fe461a0bd90" containerName="revoke-keys"
Oct 30 13:20:41.146451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.146442    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d50d47e-e814-460f-a358-5fe461a0bd90" containerName="revoke-keys"
Oct 30 13:20:41.150169 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.150151    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.166843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.166810    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"]
Oct 30 13:20:41.173517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.173487    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:20:41.173517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.173487    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:20:41.173770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.173503    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:20:41.176298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.176280    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:20:41.215668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.215603    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.215892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.215770    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.215892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.215853    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5zkd\" (UniqueName: \"kubernetes.io/projected/c5310cca-244f-4cba-a62c-1eef3ee28420-kube-api-access-r5zkd\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.270481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.270432    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" podUID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" containerName="payload-pre-processing" containerID="cri-o://0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c" gracePeriod=25
Oct 30 13:20:41.317816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.317777    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.318045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.317849    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5zkd\" (UniqueName: \"kubernetes.io/projected/c5310cca-244f-4cba-a62c-1eef3ee28420-kube-api-access-r5zkd\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.318153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.318099    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.318808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.318737    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.321191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.321170    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.338214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.338185    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5zkd\" (UniqueName: \"kubernetes.io/projected/c5310cca-244f-4cba-a62c-1eef3ee28420-kube-api-access-r5zkd\") pod \"maas-api-revoke-sub-5940752f-jpc87\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.465048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.464953    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:41.774866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:41.774833    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"]
Oct 30 13:20:41.776204 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:20:41.776176    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5310cca_244f_4cba_a62c_1eef3ee28420.slice/crio-b5703e54d9535d0f1edee87eed1e9ac0ae31ccc92f419802f7bac21c707053fb WatchSource:0}: Error finding container b5703e54d9535d0f1edee87eed1e9ac0ae31ccc92f419802f7bac21c707053fb: Status 404 returned error can't find the container with id b5703e54d9535d0f1edee87eed1e9ac0ae31ccc92f419802f7bac21c707053fb
Oct 30 13:20:42.054114 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:42.054082    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" podUID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:42.450030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:42.449987    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87" event={"ID":"c5310cca-244f-4cba-a62c-1eef3ee28420","Type":"ContainerStarted","Data":"427b3944b2e411aa6d3be8f7aeb6801f802822dff7f9fa165dbd7b1665e6d10b"}
Oct 30 13:20:42.450529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:42.450064    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87" event={"ID":"c5310cca-244f-4cba-a62c-1eef3ee28420","Type":"ContainerStarted","Data":"b5703e54d9535d0f1edee87eed1e9ac0ae31ccc92f419802f7bac21c707053fb"}
Oct 30 13:20:42.491186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:42.491129    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87" podStartSLOduration=1.491115068 podStartE2EDuration="1.491115068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:20:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:20:42.48743702 +0000 UTC m=+1052.747675253" watchObservedRunningTime="2026-10-30 13:20:42.491115068 +0000 UTC m=+1052.751353301"
Oct 30 13:20:43.459412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:43.459378    4291 generic.go:356] "Generic (PLEG): container finished" podID="c5310cca-244f-4cba-a62c-1eef3ee28420" containerID="427b3944b2e411aa6d3be8f7aeb6801f802822dff7f9fa165dbd7b1665e6d10b" exitCode=0
Oct 30 13:20:43.459822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:43.459463    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87" event={"ID":"c5310cca-244f-4cba-a62c-1eef3ee28420","Type":"ContainerDied","Data":"427b3944b2e411aa6d3be8f7aeb6801f802822dff7f9fa165dbd7b1665e6d10b"}
Oct 30 13:20:44.631866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.631839    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:44.765575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.765499    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-xks-ca\") pod \"c5310cca-244f-4cba-a62c-1eef3ee28420\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") "
Oct 30 13:20:44.765575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.765566    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5310cca-244f-4cba-a62c-1eef3ee28420-kube-api-access-r5zkd\" (UniqueName: \"kubernetes.io/projected/c5310cca-244f-4cba-a62c-1eef3ee28420-kube-api-access-r5zkd\") pod \"c5310cca-244f-4cba-a62c-1eef3ee28420\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") "
Oct 30 13:20:44.765774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.765702    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-openshift-ca\") pod \"c5310cca-244f-4cba-a62c-1eef3ee28420\" (UID: \"c5310cca-244f-4cba-a62c-1eef3ee28420\") "
Oct 30 13:20:44.766063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.766039    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-openshift-ca" pod "c5310cca-244f-4cba-a62c-1eef3ee28420" (UID: "c5310cca-244f-4cba-a62c-1eef3ee28420"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:20:44.767519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.767497    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-xks-ca" pod "c5310cca-244f-4cba-a62c-1eef3ee28420" (UID: "c5310cca-244f-4cba-a62c-1eef3ee28420"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:20:44.767667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.767642    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5310cca-244f-4cba-a62c-1eef3ee28420-kube-api-access-r5zkd" pod "c5310cca-244f-4cba-a62c-1eef3ee28420" (UID: "c5310cca-244f-4cba-a62c-1eef3ee28420"). InnerVolumeSpecName "kube-api-access-r5zkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:20:44.867591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.867554    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:44.867591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.867585    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5zkd\" (UniqueName: \"kubernetes.io/projected/c5310cca-244f-4cba-a62c-1eef3ee28420-kube-api-access-r5zkd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:44.867591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:44.867595    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c5310cca-244f-4cba-a62c-1eef3ee28420-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:20:45.475802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:45.475763    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87" event={"ID":"c5310cca-244f-4cba-a62c-1eef3ee28420","Type":"ContainerDied","Data":"b5703e54d9535d0f1edee87eed1e9ac0ae31ccc92f419802f7bac21c707053fb"}
Oct 30 13:20:45.475802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:45.475795    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"
Oct 30 13:20:45.475802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:45.475821    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5703e54d9535d0f1edee87eed1e9ac0ae31ccc92f419802f7bac21c707053fb"
Oct 30 13:20:52.054696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:52.054615    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" podUID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:20:59.865626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:20:59.862973    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:21:01.045711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.045673    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"]
Oct 30 13:21:01.110753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.110713    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-38b3eed7-fqmrt"]
Oct 30 13:21:01.465984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.465946    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"]
Oct 30 13:21:01.475465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.475432    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-zzpk7"]
Oct 30 13:21:01.569382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.569350    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:21:01.641260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.641157    4291 generic.go:356] "Generic (PLEG): container finished" podID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" containerID="0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c" exitCode=0
Oct 30 13:21:01.641260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.641219    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"
Oct 30 13:21:01.641489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.641252    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" event={"ID":"e47c1e82-2cff-4c8d-b702-05da93c3ba41","Type":"ContainerDied","Data":"0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c"}
Oct 30 13:21:01.641489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.641347    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l" event={"ID":"e47c1e82-2cff-4c8d-b702-05da93c3ba41","Type":"ContainerDied","Data":"afe891e219deebdeb129a8b7bc19a2b90870f65f8e106ebdad03450152996255"}
Oct 30 13:21:01.641489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.641382    4291 scope.go:122] "RemoveContainer" containerID="0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c"
Oct 30 13:21:01.655422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.655395    4291 scope.go:122] "RemoveContainer" containerID="0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c"
Oct 30 13:21:01.655878 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:21:01.655848    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c\": container with ID starting with 0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c not found: ID does not exist" containerID="0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c"
Oct 30 13:21:01.660153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.660050    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c"} err="failed to get container status \"0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c\": rpc error: code = NotFound desc = could not find container \"0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c\": container with ID starting with 0d12c29f9fe43d3de79fb9f9b9717ea805e037d8adb72a05d2b4ce4b7d63368c not found: ID does not exist"
Oct 30 13:21:01.758275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.758244    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e47c1e82-2cff-4c8d-b702-05da93c3ba41-kube-api-access-hdlp6\" (UniqueName: \"kubernetes.io/projected/e47c1e82-2cff-4c8d-b702-05da93c3ba41-kube-api-access-hdlp6\") pod \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") "
Oct 30 13:21:01.758471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.758322    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e47c1e82-2cff-4c8d-b702-05da93c3ba41-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47c1e82-2cff-4c8d-b702-05da93c3ba41-plugins-config-volume\") pod \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\" (UID: \"e47c1e82-2cff-4c8d-b702-05da93c3ba41\") "
Oct 30 13:21:01.758813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.758785    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e47c1e82-2cff-4c8d-b702-05da93c3ba41-plugins-config-volume" pod "e47c1e82-2cff-4c8d-b702-05da93c3ba41" (UID: "e47c1e82-2cff-4c8d-b702-05da93c3ba41"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:01.760657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.760614    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e47c1e82-2cff-4c8d-b702-05da93c3ba41-kube-api-access-hdlp6" pod "e47c1e82-2cff-4c8d-b702-05da93c3ba41" (UID: "e47c1e82-2cff-4c8d-b702-05da93c3ba41"). InnerVolumeSpecName "kube-api-access-hdlp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:01.860204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.860167    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdlp6\" (UniqueName: \"kubernetes.io/projected/e47c1e82-2cff-4c8d-b702-05da93c3ba41-kube-api-access-hdlp6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:01.860204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.860198    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e47c1e82-2cff-4c8d-b702-05da93c3ba41-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:01.888925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.888885    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7204027e-9bf3-44fd-b9b0-9726409daefb" path="/var/lib/kubelet/pods/7204027e-9bf3-44fd-b9b0-9726409daefb/volumes"
Oct 30 13:21:01.889548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.889521    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa9099ad-07db-440c-b4b6-f0e1d71e4e9e" path="/var/lib/kubelet/pods/aa9099ad-07db-440c-b4b6-f0e1d71e4e9e/volumes"
Oct 30 13:21:01.991678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:01.991615    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"]
Oct 30 13:21:02.001172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:02.001141    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6b7b93a3-797d7cbf99-8h84l"]
Oct 30 13:21:03.878840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:03.878808    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" path="/var/lib/kubelet/pods/e47c1e82-2cff-4c8d-b702-05da93c3ba41/volumes"
Oct 30 13:21:08.211171 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:21:08.211133    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:21:10.251874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.251833    4291 scope.go:122] "RemoveContainer" containerID="ce78ab0bf40ecd3224fe194f721cdd5e02c1fdb6de38cd002b2e26e539891cf5"
Oct 30 13:21:10.265918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.265886    4291 scope.go:122] "RemoveContainer" containerID="fd5ca7ef7c1dec145f62f21a81a9915083ebfc88610e7d1659e419d1219f8b4a"
Oct 30 13:21:10.279891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.279857    4291 scope.go:122] "RemoveContainer" containerID="192b0485fa8e232beb227c89feda6a751050b8f4cace74fa9f03d56cabce9e7a"
Oct 30 13:21:10.294256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.294042    4291 scope.go:122] "RemoveContainer" containerID="922f26f493504a2584df7800564a36faebc280c2416ad289da2d50a030a435cc"
Oct 30 13:21:10.307458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.307421    4291 scope.go:122] "RemoveContainer" containerID="4e6dcad551c70b92ac7e28c7245cea2d69ee5dd34cfe85ab3258351f3ff956f8"
Oct 30 13:21:10.323347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.323323    4291 scope.go:122] "RemoveContainer" containerID="a436e5961f005cec9e3ea63464037cdf14217f448c4adaaca9815244a1df8e74"
Oct 30 13:21:10.334544 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.334518    4291 scope.go:122] "RemoveContainer" containerID="acc185fecb5b93b7b5553934f8211f9b9cf84afd38bd836a5c861f73696d2fd6"
Oct 30 13:21:10.618972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.618934    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"]
Oct 30 13:21:10.629089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:10.629058    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2ed395-dcrwk"]
Oct 30 13:21:11.880350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:11.880312    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d50d47e-e814-460f-a358-5fe461a0bd90" path="/var/lib/kubelet/pods/4d50d47e-e814-460f-a358-5fe461a0bd90/volumes"
Oct 30 13:21:13.031168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:13.031134    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"]
Oct 30 13:21:13.043055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:13.043021    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7f6cce00-hhd2g"]
Oct 30 13:21:13.881767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:13.881729    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b398c60-0b41-4ecc-80d3-2e336559a338" path="/var/lib/kubelet/pods/8b398c60-0b41-4ecc-80d3-2e336559a338/volumes"
Oct 30 13:21:21.634727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.634693    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:21:21.734922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.734884    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"]
Oct 30 13:21:21.735774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.735745    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c5310cca-244f-4cba-a62c-1eef3ee28420" containerName="revoke-keys"
Oct 30 13:21:21.735774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.735772    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5310cca-244f-4cba-a62c-1eef3ee28420" containerName="revoke-keys"
Oct 30 13:21:21.735975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.735844    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" containerName="payload-pre-processing"
Oct 30 13:21:21.735975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.735854    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" containerName="payload-pre-processing"
Oct 30 13:21:21.736089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.735989    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e47c1e82-2cff-4c8d-b702-05da93c3ba41" containerName="payload-pre-processing"
Oct 30 13:21:21.736089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.736009    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5310cca-244f-4cba-a62c-1eef3ee28420" containerName="revoke-keys"
Oct 30 13:21:21.753705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.747740    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.772233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.770217    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:21:21.772233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.770913    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:21:21.798600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.794141    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"]
Oct 30 13:21:21.863708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.863659    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.863891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.863724    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.864242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.864205    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zthhl\" (UniqueName: \"kubernetes.io/projected/63f29e84-8029-4d3d-8956-51937104d2d0-kube-api-access-zthhl\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.868724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.868701    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:21:21.868949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.868835    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:21:21.966201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.966067    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.966780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.966758    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zthhl\" (UniqueName: \"kubernetes.io/projected/63f29e84-8029-4d3d-8956-51937104d2d0-kube-api-access-zthhl\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.967707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.966866    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.967707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.966888    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.971675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.970531    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:21.986353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:21.986322    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zthhl\" (UniqueName: \"kubernetes.io/projected/63f29e84-8029-4d3d-8956-51937104d2d0-kube-api-access-zthhl\") pod \"maas-api-revoke-sub-20924782-8cwch\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:22.052058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:22.052023    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"]
Oct 30 13:21:22.061365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:22.061337    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-555de3ea-275bv"]
Oct 30 13:21:22.093890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:22.093859    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:22.398006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:22.397973    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"]
Oct 30 13:21:22.399830 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:21:22.399802    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63f29e84_8029_4d3d_8956_51937104d2d0.slice/crio-aad46f7038220616555457803dba2479e4d26e7c041e6cc8b3da0e15625e9c93 WatchSource:0}: Error finding container aad46f7038220616555457803dba2479e4d26e7c041e6cc8b3da0e15625e9c93: Status 404 returned error can't find the container with id aad46f7038220616555457803dba2479e4d26e7c041e6cc8b3da0e15625e9c93
Oct 30 13:21:22.849284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:22.849251    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch" event={"ID":"63f29e84-8029-4d3d-8956-51937104d2d0","Type":"ContainerStarted","Data":"f9d749396163160285f17a560d6b1f31fcd4e1083861826ccc407d65d3345007"}
Oct 30 13:21:22.849774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:22.849316    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch" event={"ID":"63f29e84-8029-4d3d-8956-51937104d2d0","Type":"ContainerStarted","Data":"aad46f7038220616555457803dba2479e4d26e7c041e6cc8b3da0e15625e9c93"}
Oct 30 13:21:22.896999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:22.895355    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch" podStartSLOduration=1.895335183 podStartE2EDuration="1.895335183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:22.894001329 +0000 UTC m=+1093.154239565" watchObservedRunningTime="2026-10-30 13:21:22.895335183 +0000 UTC m=+1093.155573419"
Oct 30 13:21:23.858595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:23.858555    4291 generic.go:356] "Generic (PLEG): container finished" podID="63f29e84-8029-4d3d-8956-51937104d2d0" containerID="f9d749396163160285f17a560d6b1f31fcd4e1083861826ccc407d65d3345007" exitCode=0
Oct 30 13:21:23.859069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:23.858652    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch" event={"ID":"63f29e84-8029-4d3d-8956-51937104d2d0","Type":"ContainerDied","Data":"f9d749396163160285f17a560d6b1f31fcd4e1083861826ccc407d65d3345007"}
Oct 30 13:21:23.878837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:23.878808    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63c9f3c4-b806-48aa-8e30-c64719a98c06" path="/var/lib/kubelet/pods/63c9f3c4-b806-48aa-8e30-c64719a98c06/volumes"
Oct 30 13:21:25.020839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.020808    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:25.114505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.114462    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/63f29e84-8029-4d3d-8956-51937104d2d0-kube-api-access-zthhl\" (UniqueName: \"kubernetes.io/projected/63f29e84-8029-4d3d-8956-51937104d2d0-kube-api-access-zthhl\") pod \"63f29e84-8029-4d3d-8956-51937104d2d0\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") "
Oct 30 13:21:25.114802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.114782    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-openshift-ca\") pod \"63f29e84-8029-4d3d-8956-51937104d2d0\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") "
Oct 30 13:21:25.114854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.114840    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-xks-ca\") pod \"63f29e84-8029-4d3d-8956-51937104d2d0\" (UID: \"63f29e84-8029-4d3d-8956-51937104d2d0\") "
Oct 30 13:21:25.115153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.115125    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-openshift-ca" pod "63f29e84-8029-4d3d-8956-51937104d2d0" (UID: "63f29e84-8029-4d3d-8956-51937104d2d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:25.115394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.115376    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:25.116896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.116828    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/63f29e84-8029-4d3d-8956-51937104d2d0-kube-api-access-zthhl" pod "63f29e84-8029-4d3d-8956-51937104d2d0" (UID: "63f29e84-8029-4d3d-8956-51937104d2d0"). InnerVolumeSpecName "kube-api-access-zthhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:25.117011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.116889    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-xks-ca" pod "63f29e84-8029-4d3d-8956-51937104d2d0" (UID: "63f29e84-8029-4d3d-8956-51937104d2d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:25.216607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.216564    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/63f29e84-8029-4d3d-8956-51937104d2d0-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:25.216607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.216601    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zthhl\" (UniqueName: \"kubernetes.io/projected/63f29e84-8029-4d3d-8956-51937104d2d0-kube-api-access-zthhl\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:25.875229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.875192    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"
Oct 30 13:21:25.882328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.882281    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch" event={"ID":"63f29e84-8029-4d3d-8956-51937104d2d0","Type":"ContainerDied","Data":"aad46f7038220616555457803dba2479e4d26e7c041e6cc8b3da0e15625e9c93"}
Oct 30 13:21:25.882510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:25.882346    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad46f7038220616555457803dba2479e4d26e7c041e6cc8b3da0e15625e9c93"
Oct 30 13:21:27.784821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.784783    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"]
Oct 30 13:21:27.785371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.785355    4291 cpu_manager.go:414] "containerMap: removing container" podUID="63f29e84-8029-4d3d-8956-51937104d2d0" containerName="revoke-keys"
Oct 30 13:21:27.785436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.785372    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="63f29e84-8029-4d3d-8956-51937104d2d0" containerName="revoke-keys"
Oct 30 13:21:27.785472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.785453    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="63f29e84-8029-4d3d-8956-51937104d2d0" containerName="revoke-keys"
Oct 30 13:21:27.797489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.797448    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:27.811056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.811022    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"]
Oct 30 13:21:27.813215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.813165    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-8ac28e\""
Oct 30 13:21:27.814786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.814762    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-8ac28e-dockercfg-q8f2q\""
Oct 30 13:21:27.864241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.864194    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a22c44-e47f-43a3-ac45-90ff7853e5ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:27.864449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.864338    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5sppp\" (UniqueName: \"kubernetes.io/projected/21a22c44-e47f-43a3-ac45-90ff7853e5ba-kube-api-access-5sppp\") pod \"payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:27.966056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.966013    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a22c44-e47f-43a3-ac45-90ff7853e5ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:27.966347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.966328    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5sppp\" (UniqueName: \"kubernetes.io/projected/21a22c44-e47f-43a3-ac45-90ff7853e5ba-kube-api-access-5sppp\") pod \"payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:27.970969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.970938    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a22c44-e47f-43a3-ac45-90ff7853e5ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:27.986767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:27.986737    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5sppp\" (UniqueName: \"kubernetes.io/projected/21a22c44-e47f-43a3-ac45-90ff7853e5ba-kube-api-access-5sppp\") pod \"payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:28.120081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:28.119550    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:28.446223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:28.446188    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"]
Oct 30 13:21:28.447112 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:21:28.447079    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a22c44_e47f_43a3_ac45_90ff7853e5ba.slice/crio-4927e1cae3634c6ae44e243c514b9716e442f439566f2cab579e7c8ce62693c8 WatchSource:0}: Error finding container 4927e1cae3634c6ae44e243c514b9716e442f439566f2cab579e7c8ce62693c8: Status 404 returned error can't find the container with id 4927e1cae3634c6ae44e243c514b9716e442f439566f2cab579e7c8ce62693c8
Oct 30 13:21:28.914665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:28.914604    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l" event={"ID":"21a22c44-e47f-43a3-ac45-90ff7853e5ba","Type":"ContainerStarted","Data":"4927e1cae3634c6ae44e243c514b9716e442f439566f2cab579e7c8ce62693c8"}
Oct 30 13:21:30.932395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:30.932356    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l" event={"ID":"21a22c44-e47f-43a3-ac45-90ff7853e5ba","Type":"ContainerStarted","Data":"d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12"}
Oct 30 13:21:30.932987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:30.932551    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:30.971797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:30.971744    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l" podStartSLOduration=2.431600757 podStartE2EDuration="3.971726833s" totalImagesPullingTime="1.540126076s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:30.970621745 +0000 UTC m=+1101.230859981" watchObservedRunningTime="2026-10-30 13:21:30.971726833 +0000 UTC m=+1101.231965065"
Oct 30 13:21:37.042900 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:37.042860    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"]
Oct 30 13:21:37.054358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:37.054323    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24428ba5-ltj5m"]
Oct 30 13:21:37.914852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:37.914458    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d71bfbe-492a-40be-a378-f80a80d22772" path="/var/lib/kubelet/pods/7d71bfbe-492a-40be-a378-f80a80d22772/volumes"
Oct 30 13:21:38.229330 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:21:38.229228    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:21:39.233660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.233466    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"]
Oct 30 13:21:39.241583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.241546    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.244204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.244176    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ztht\" (UniqueName: \"kubernetes.io/projected/43fc1425-d02d-4f14-bc76-297492f67a91-kube-api-access-8ztht\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.244580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.244559    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc1425-d02d-4f14-bc76-297492f67a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.245285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.245251    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"]
Oct 30 13:21:39.259778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.259600    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dcf84\""
Oct 30 13:21:39.273705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.272628    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dcf84-dockercfg-pf7qd\""
Oct 30 13:21:39.355772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.355728    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ztht\" (UniqueName: \"kubernetes.io/projected/43fc1425-d02d-4f14-bc76-297492f67a91-kube-api-access-8ztht\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.404692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.380038    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc1425-d02d-4f14-bc76-297492f67a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.521821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.518264    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"]
Oct 30 13:21:39.526363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.526306    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc1425-d02d-4f14-bc76-297492f67a91-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.566743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.546031    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:39.566743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.559933    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"]
Oct 30 13:21:39.624660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.622418    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ztht\" (UniqueName: \"kubernetes.io/projected/43fc1425-d02d-4f14-bc76-297492f67a91-kube-api-access-8ztht\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.624660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.624454    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-kube-api-access-csl6r\") pod \"payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:39.638698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.624627    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:39.721658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.720702    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:39.738713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.738678    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-kube-api-access-csl6r\") pod \"payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:39.738894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.738849    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:39.742600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.739912    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:39.788698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.788571    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-kube-api-access-csl6r\") pod \"payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:39.867663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:39.844624    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:40.894943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:40.883037    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"]
Oct 30 13:21:41.047557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:41.047515    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"]
Oct 30 13:21:41.162092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:41.162053    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj" event={"ID":"43fc1425-d02d-4f14-bc76-297492f67a91","Type":"ContainerStarted","Data":"7b72813d18269e1ee4a465a914af5c8b559f7827189b23606137c217c2863520"}
Oct 30 13:21:41.164126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:41.164075    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd" event={"ID":"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95","Type":"ContainerStarted","Data":"f977143b2feda3859debc9edcad28c6c7d8ab22ce2ed3d3516d54ee1517034ce"}
Oct 30 13:21:41.942876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:41.942847    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:21:42.173727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:42.173679    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj" event={"ID":"43fc1425-d02d-4f14-bc76-297492f67a91","Type":"ContainerStarted","Data":"398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac"}
Oct 30 13:21:42.173968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:42.173762    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:42.177279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:42.176326    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd" event={"ID":"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95","Type":"ContainerStarted","Data":"d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef"}
Oct 30 13:21:42.177279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:42.177218    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:42.215383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:42.215270    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj" podStartSLOduration=3.215248583 podStartE2EDuration="3.215248583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:42.211309242 +0000 UTC m=+1112.471547477" watchObservedRunningTime="2026-10-30 13:21:42.215248583 +0000 UTC m=+1112.475486821"
Oct 30 13:21:42.251148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:42.251093    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd" podStartSLOduration=3.251077979 podStartE2EDuration="3.251077979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:42.248659668 +0000 UTC m=+1112.508897901" watchObservedRunningTime="2026-10-30 13:21:42.251077979 +0000 UTC m=+1112.511316244"
Oct 30 13:21:47.457959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.457731    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"]
Oct 30 13:21:47.469047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.463195    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.497348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.497314    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"]
Oct 30 13:21:47.497793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.497762    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-121d75-openshift-default-dockercfg-6b2z8\""
Oct 30 13:21:47.597479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.597439    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-credential-socket\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.597685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.597661    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-podinfo\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.597748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.597710    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xth85\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-kube-api-access-xth85\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.597808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.597789    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istiod-ca-cert\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.597894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.597872    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-certs\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.597989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.597975    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-token\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.598067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.598055    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-socket\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.598105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.598077    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-envoy\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.598159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.598106    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-data\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702441    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-credential-socket\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702577    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-podinfo\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702614    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xth85\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-kube-api-access-xth85\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702663    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istiod-ca-cert\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702726    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-certs\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702822    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-token\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702886    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-socket\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702915    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-envoy\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.702938    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-data\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.703660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.703446    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-data\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.717711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.704673    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istiod-ca-cert\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.717711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.704930    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-socket\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.717711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.705773    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-credential-socket\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.717711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.705997    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-certs\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.735653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.731976    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-envoy\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.739528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.739218    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xth85\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-kube-api-access-xth85\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.739528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.739476    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-token\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.756662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.751512    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-podinfo\") pod \"e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") " pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:47.793234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:47.792803    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:48.166373 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:21:48.166343    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod533c0e6f_0446_40d9_b4fa_58fbd1c34c59.slice/crio-ae41379bbd217069630602c4eb40873aa81ce059d6a87c663cdf75f0d4bdd572 WatchSource:0}: Error finding container ae41379bbd217069630602c4eb40873aa81ce059d6a87c663cdf75f0d4bdd572: Status 404 returned error can't find the container with id ae41379bbd217069630602c4eb40873aa81ce059d6a87c663cdf75f0d4bdd572
Oct 30 13:21:48.167797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:48.167764    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"]
Oct 30 13:21:48.169413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:48.169368    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:48.169511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:48.169460    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:48.169511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:48.169490    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:21:48.230708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:48.230676    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" event={"ID":"533c0e6f-0446-40d9-b4fa-58fbd1c34c59","Type":"ContainerStarted","Data":"ae41379bbd217069630602c4eb40873aa81ce059d6a87c663cdf75f0d4bdd572"}
Oct 30 13:21:49.206810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.206767    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"]
Oct 30 13:21:49.211317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.211293    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.234134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.234099    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"]
Oct 30 13:21:49.239692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.239662    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-1f486a\""
Oct 30 13:21:49.245114 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.245080    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" event={"ID":"533c0e6f-0446-40d9-b4fa-58fbd1c34c59","Type":"ContainerStarted","Data":"ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34"}
Oct 30 13:21:49.256216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.256190    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-1f486a-dockercfg-vm55x\""
Oct 30 13:21:49.270867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.270826    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"]
Oct 30 13:21:49.275913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.275891    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.296527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.296494    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"]
Oct 30 13:21:49.306009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.305917    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" podStartSLOduration=2.305899915 podStartE2EDuration="2.305899915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:49.303575918 +0000 UTC m=+1119.563814152" watchObservedRunningTime="2026-10-30 13:21:49.305899915 +0000 UTC m=+1119.566138151"
Oct 30 13:21:49.358658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.358398    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrz7b\" (UniqueName: \"kubernetes.io/projected/676e64fd-59d0-4d95-819e-981d5d35d7e0-kube-api-access-rrz7b\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.358658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.358600    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e64fd-59d0-4d95-819e-981d5d35d7e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.358925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.358672    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2221ef3b-2cbe-49b4-a944-c80c14a051f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1f486a-568c587754-95mld\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.358925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.358721    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwmt7\" (UniqueName: \"kubernetes.io/projected/2221ef3b-2cbe-49b4-a944-c80c14a051f4-kube-api-access-zwmt7\") pod \"payload-processing-e2e-ipp-b-1f486a-568c587754-95mld\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.461709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.460296    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrz7b\" (UniqueName: \"kubernetes.io/projected/676e64fd-59d0-4d95-819e-981d5d35d7e0-kube-api-access-rrz7b\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.461709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.460489    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e64fd-59d0-4d95-819e-981d5d35d7e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.461709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.460544    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2221ef3b-2cbe-49b4-a944-c80c14a051f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1f486a-568c587754-95mld\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.461709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.460590    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwmt7\" (UniqueName: \"kubernetes.io/projected/2221ef3b-2cbe-49b4-a944-c80c14a051f4-kube-api-access-zwmt7\") pod \"payload-processing-e2e-ipp-b-1f486a-568c587754-95mld\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.462067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.461851    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e64fd-59d0-4d95-819e-981d5d35d7e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.462756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.462450    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2221ef3b-2cbe-49b4-a944-c80c14a051f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-1f486a-568c587754-95mld\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.480168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.480132    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrz7b\" (UniqueName: \"kubernetes.io/projected/676e64fd-59d0-4d95-819e-981d5d35d7e0-kube-api-access-rrz7b\") pod \"payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.480691 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.480657    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwmt7\" (UniqueName: \"kubernetes.io/projected/2221ef3b-2cbe-49b4-a944-c80c14a051f4-kube-api-access-zwmt7\") pod \"payload-processing-e2e-ipp-b-1f486a-568c587754-95mld\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.531774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.531712    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:49.591312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.591274    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:49.794673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.794566    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:49.805505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.805158    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:49.867097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.866457    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"]
Oct 30 13:21:49.982012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:49.981980    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"]
Oct 30 13:21:49.983309 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:21:49.983282    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676e64fd_59d0_4d95_819e_981d5d35d7e0.slice/crio-256d4ade5b8abd41f414c1d583c60496cf3b4a83adac3243cbcd4f7f4332a1cd WatchSource:0}: Error finding container 256d4ade5b8abd41f414c1d583c60496cf3b4a83adac3243cbcd4f7f4332a1cd: Status 404 returned error can't find the container with id 256d4ade5b8abd41f414c1d583c60496cf3b4a83adac3243cbcd4f7f4332a1cd
Oct 30 13:21:50.033312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.033279    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"]
Oct 30 13:21:50.039737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.039712    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44dfaa8b-ckpb2"]
Oct 30 13:21:50.259446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.258591    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld" event={"ID":"2221ef3b-2cbe-49b4-a944-c80c14a051f4","Type":"ContainerStarted","Data":"1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed"}
Oct 30 13:21:50.259446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.258690    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld" event={"ID":"2221ef3b-2cbe-49b4-a944-c80c14a051f4","Type":"ContainerStarted","Data":"63ea9db8b446a143aa23283cec882b8077180f1651b7b36eec9d88ccb39965ef"}
Oct 30 13:21:50.259446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.258927    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:21:50.261026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.260992    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4" event={"ID":"676e64fd-59d0-4d95-819e-981d5d35d7e0","Type":"ContainerStarted","Data":"89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c"}
Oct 30 13:21:50.261222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.261053    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4" event={"ID":"676e64fd-59d0-4d95-819e-981d5d35d7e0","Type":"ContainerStarted","Data":"256d4ade5b8abd41f414c1d583c60496cf3b4a83adac3243cbcd4f7f4332a1cd"}
Oct 30 13:21:50.261316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.261296    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:50.261400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.261350    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:21:50.262597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.262572    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:21:50.297511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.297456    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld" podStartSLOduration=1.297441258 podStartE2EDuration="1.297441258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:50.295867441 +0000 UTC m=+1120.556105675" watchObservedRunningTime="2026-10-30 13:21:50.297441258 +0000 UTC m=+1120.557679492"
Oct 30 13:21:50.381575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.381507    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4" podStartSLOduration=1.3814828000000001 podStartE2EDuration="1.3814828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:50.375007111 +0000 UTC m=+1120.635245380" watchObservedRunningTime="2026-10-30 13:21:50.3814828 +0000 UTC m=+1120.641721036"
Oct 30 13:21:50.565577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.565497    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"]
Oct 30 13:21:50.576548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.576489    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.596237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.594404    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:21:50.596237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.594734    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:21:50.596237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.595168    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:21:50.596237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.595485    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:21:50.606804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.603059    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"]
Oct 30 13:21:50.706432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.706390    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.706617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.706552    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6plwr\" (UniqueName: \"kubernetes.io/projected/08066830-1005-43ef-9c8a-2a4bd4aa1c48-kube-api-access-6plwr\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.706617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.706596    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.808149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.808114    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.808336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.808204    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6plwr\" (UniqueName: \"kubernetes.io/projected/08066830-1005-43ef-9c8a-2a4bd4aa1c48-kube-api-access-6plwr\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.808336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.808229    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.808811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.808786    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.810532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.810514    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.827156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.827089    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6plwr\" (UniqueName: \"kubernetes.io/projected/08066830-1005-43ef-9c8a-2a4bd4aa1c48-kube-api-access-6plwr\") pod \"maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:50.914308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:50.914275    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:51.141718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.141687    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"]
Oct 30 13:21:51.147057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.147039    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.162909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.162872    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"]
Oct 30 13:21:51.213719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.213684    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.213886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.213798    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s99j6\" (UniqueName: \"kubernetes.io/projected/0e125e83-56b5-4168-a1a5-40bae1600fad-kube-api-access-s99j6\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.213886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.213874    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.226786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.226755    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"]
Oct 30 13:21:51.227423 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:21:51.227398    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08066830_1005_43ef_9c8a_2a4bd4aa1c48.slice/crio-61319013daf865858d7429cd2796b510ba45eab35f64e06906505a1cdb1e92ea WatchSource:0}: Error finding container 61319013daf865858d7429cd2796b510ba45eab35f64e06906505a1cdb1e92ea: Status 404 returned error can't find the container with id 61319013daf865858d7429cd2796b510ba45eab35f64e06906505a1cdb1e92ea
Oct 30 13:21:51.271923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.271882    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp" event={"ID":"08066830-1005-43ef-9c8a-2a4bd4aa1c48","Type":"ContainerStarted","Data":"61319013daf865858d7429cd2796b510ba45eab35f64e06906505a1cdb1e92ea"}
Oct 30 13:21:51.315214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.315176    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.315414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.315391    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s99j6\" (UniqueName: \"kubernetes.io/projected/0e125e83-56b5-4168-a1a5-40bae1600fad-kube-api-access-s99j6\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.315584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.315556    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.316238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.316213    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.317716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.317697    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.334238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.334211    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s99j6\" (UniqueName: \"kubernetes.io/projected/0e125e83-56b5-4168-a1a5-40bae1600fad-kube-api-access-s99j6\") pod \"maas-api-revoke-keys-e2e-api-a-3c2202-l4frb\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.465563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.465483    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:51.770806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.770777    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"]
Oct 30 13:21:51.771454 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:21:51.771432    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e125e83_56b5_4168_a1a5_40bae1600fad.slice/crio-235c7542a1642b9ac27297fdd8ae6c6ad115a1f77ebde617a91ce91cd05b001f WatchSource:0}: Error finding container 235c7542a1642b9ac27297fdd8ae6c6ad115a1f77ebde617a91ce91cd05b001f: Status 404 returned error can't find the container with id 235c7542a1642b9ac27297fdd8ae6c6ad115a1f77ebde617a91ce91cd05b001f
Oct 30 13:21:51.880175 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:51.880143    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54cd6743-b67e-4236-a62b-2005ba1b96b7" path="/var/lib/kubelet/pods/54cd6743-b67e-4236-a62b-2005ba1b96b7/volumes"
Oct 30 13:21:52.282710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:52.282671    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb" event={"ID":"0e125e83-56b5-4168-a1a5-40bae1600fad","Type":"ContainerStarted","Data":"83542fc9082ed6e238894c6d20bee8b23c105573e67cde3d2e05890af298ede3"}
Oct 30 13:21:52.283246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:52.282748    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb" event={"ID":"0e125e83-56b5-4168-a1a5-40bae1600fad","Type":"ContainerStarted","Data":"235c7542a1642b9ac27297fdd8ae6c6ad115a1f77ebde617a91ce91cd05b001f"}
Oct 30 13:21:52.284918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:52.284866    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp" event={"ID":"08066830-1005-43ef-9c8a-2a4bd4aa1c48","Type":"ContainerStarted","Data":"4d3aef48f715f0fd937d6de32b7d3553b1d40d825e9a804ea29368365c78fa67"}
Oct 30 13:21:52.322003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:52.321886    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb" podStartSLOduration=1.321871072 podStartE2EDuration="1.321871072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:52.319002606 +0000 UTC m=+1122.579240840" watchObservedRunningTime="2026-10-30 13:21:52.321871072 +0000 UTC m=+1122.582109306"
Oct 30 13:21:52.359052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:52.359000    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp" podStartSLOduration=2.358985894 podStartE2EDuration="2.358985894s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:52.356572786 +0000 UTC m=+1122.616811022" watchObservedRunningTime="2026-10-30 13:21:52.358985894 +0000 UTC m=+1122.619224128"
Oct 30 13:21:53.187584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.187549    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:21:53.294325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.294290    4291 generic.go:356] "Generic (PLEG): container finished" podID="0e125e83-56b5-4168-a1a5-40bae1600fad" containerID="83542fc9082ed6e238894c6d20bee8b23c105573e67cde3d2e05890af298ede3" exitCode=0
Oct 30 13:21:53.294797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.294366    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb" event={"ID":"0e125e83-56b5-4168-a1a5-40bae1600fad","Type":"ContainerDied","Data":"83542fc9082ed6e238894c6d20bee8b23c105573e67cde3d2e05890af298ede3"}
Oct 30 13:21:53.296237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.296213    4291 generic.go:356] "Generic (PLEG): container finished" podID="08066830-1005-43ef-9c8a-2a4bd4aa1c48" containerID="4d3aef48f715f0fd937d6de32b7d3553b1d40d825e9a804ea29368365c78fa67" exitCode=0
Oct 30 13:21:53.296380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.296266    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp" event={"ID":"08066830-1005-43ef-9c8a-2a4bd4aa1c48","Type":"ContainerDied","Data":"4d3aef48f715f0fd937d6de32b7d3553b1d40d825e9a804ea29368365c78fa67"}
Oct 30 13:21:53.612176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.612130    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"]
Oct 30 13:21:53.616896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.616870    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:53.631248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.631192    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-121d75\""
Oct 30 13:21:53.631389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.631257    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-121d75-dockercfg-ql59n\""
Oct 30 13:21:53.633142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.633116    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"]
Oct 30 13:21:53.759658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.751602    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:53.759658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.751816    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75zwt\" (UniqueName: \"kubernetes.io/projected/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-kube-api-access-75zwt\") pod \"payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:53.854564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.854511    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:53.854773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.854586    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75zwt\" (UniqueName: \"kubernetes.io/projected/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-kube-api-access-75zwt\") pod \"payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:53.856261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.856236    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:53.877698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.876472    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75zwt\" (UniqueName: \"kubernetes.io/projected/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-kube-api-access-75zwt\") pod \"payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") " pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:53.937956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:53.937923    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:54.202875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.202800    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:21:54.267249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.267207    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"]
Oct 30 13:21:54.272534 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:21:54.272503    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b496e92_b6a4_45cc_bf94_3e7b824a0b4e.slice/crio-76d7b322245fe79b445a7e9dcc15179b14eb4b602d85dc8ed7820129ec93cfce WatchSource:0}: Error finding container 76d7b322245fe79b445a7e9dcc15179b14eb4b602d85dc8ed7820129ec93cfce: Status 404 returned error can't find the container with id 76d7b322245fe79b445a7e9dcc15179b14eb4b602d85dc8ed7820129ec93cfce
Oct 30 13:21:54.306924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.306895    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7" event={"ID":"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e","Type":"ContainerStarted","Data":"76d7b322245fe79b445a7e9dcc15179b14eb4b602d85dc8ed7820129ec93cfce"}
Oct 30 13:21:54.502848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.502812    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:54.522899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.522871    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:54.564914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.564882    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08066830-1005-43ef-9c8a-2a4bd4aa1c48-kube-api-access-6plwr\" (UniqueName: \"kubernetes.io/projected/08066830-1005-43ef-9c8a-2a4bd4aa1c48-kube-api-access-6plwr\") pod \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") "
Oct 30 13:21:54.565110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.564934    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-xks-ca\") pod \"0e125e83-56b5-4168-a1a5-40bae1600fad\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") "
Oct 30 13:21:54.565110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.564953    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e125e83-56b5-4168-a1a5-40bae1600fad-kube-api-access-s99j6\" (UniqueName: \"kubernetes.io/projected/0e125e83-56b5-4168-a1a5-40bae1600fad-kube-api-access-s99j6\") pod \"0e125e83-56b5-4168-a1a5-40bae1600fad\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") "
Oct 30 13:21:54.565110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.565035    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-openshift-ca\") pod \"0e125e83-56b5-4168-a1a5-40bae1600fad\" (UID: \"0e125e83-56b5-4168-a1a5-40bae1600fad\") "
Oct 30 13:21:54.565110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.565063    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-openshift-ca\") pod \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") "
Oct 30 13:21:54.565110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.565100    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-xks-ca\") pod \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\" (UID: \"08066830-1005-43ef-9c8a-2a4bd4aa1c48\") "
Oct 30 13:21:54.566191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.566150    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-openshift-ca" pod "08066830-1005-43ef-9c8a-2a4bd4aa1c48" (UID: "08066830-1005-43ef-9c8a-2a4bd4aa1c48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:54.566395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.566369    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-openshift-ca" pod "0e125e83-56b5-4168-a1a5-40bae1600fad" (UID: "0e125e83-56b5-4168-a1a5-40bae1600fad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:21:54.567760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.567726    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e125e83-56b5-4168-a1a5-40bae1600fad-kube-api-access-s99j6" pod "0e125e83-56b5-4168-a1a5-40bae1600fad" (UID: "0e125e83-56b5-4168-a1a5-40bae1600fad"). InnerVolumeSpecName "kube-api-access-s99j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:54.567863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.567773    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08066830-1005-43ef-9c8a-2a4bd4aa1c48-kube-api-access-6plwr" pod "08066830-1005-43ef-9c8a-2a4bd4aa1c48" (UID: "08066830-1005-43ef-9c8a-2a4bd4aa1c48"). InnerVolumeSpecName "kube-api-access-6plwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:21:54.567919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.567893    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-xks-ca" pod "0e125e83-56b5-4168-a1a5-40bae1600fad" (UID: "0e125e83-56b5-4168-a1a5-40bae1600fad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:54.567980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.567960    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-xks-ca" pod "08066830-1005-43ef-9c8a-2a4bd4aa1c48" (UID: "08066830-1005-43ef-9c8a-2a4bd4aa1c48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:21:54.666892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.666851    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6plwr\" (UniqueName: \"kubernetes.io/projected/08066830-1005-43ef-9c8a-2a4bd4aa1c48-kube-api-access-6plwr\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:54.666892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.666887    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:54.666892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.666897    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s99j6\" (UniqueName: \"kubernetes.io/projected/0e125e83-56b5-4168-a1a5-40bae1600fad-kube-api-access-s99j6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:54.666892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.666905    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e125e83-56b5-4168-a1a5-40bae1600fad-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:54.667162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.666914    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:54.667162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:54.666922    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08066830-1005-43ef-9c8a-2a4bd4aa1c48-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:21:55.317594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.317495    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"
Oct 30 13:21:55.317594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.317522    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp" event={"ID":"08066830-1005-43ef-9c8a-2a4bd4aa1c48","Type":"ContainerDied","Data":"61319013daf865858d7429cd2796b510ba45eab35f64e06906505a1cdb1e92ea"}
Oct 30 13:21:55.318046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.317677    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="61319013daf865858d7429cd2796b510ba45eab35f64e06906505a1cdb1e92ea"
Oct 30 13:21:55.323403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.323379    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"
Oct 30 13:21:55.323403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.323394    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb" event={"ID":"0e125e83-56b5-4168-a1a5-40bae1600fad","Type":"ContainerDied","Data":"235c7542a1642b9ac27297fdd8ae6c6ad115a1f77ebde617a91ce91cd05b001f"}
Oct 30 13:21:55.323680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.323434    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="235c7542a1642b9ac27297fdd8ae6c6ad115a1f77ebde617a91ce91cd05b001f"
Oct 30 13:21:55.325730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.325706    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7" event={"ID":"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e","Type":"ContainerStarted","Data":"4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5"}
Oct 30 13:21:55.325841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.325821    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:21:55.373852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:55.373805    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7" podStartSLOduration=2.373790926 podStartE2EDuration="2.373790926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:21:55.362885197 +0000 UTC m=+1125.623123466" watchObservedRunningTime="2026-10-30 13:21:55.373790926 +0000 UTC m=+1125.634029159"
Oct 30 13:21:56.202626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:21:56.202592    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:22:01.275484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:01.275456    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:22:01.275896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:01.275524    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:22:03.031823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:03.031788    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"]
Oct 30 13:22:03.043153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:03.043124    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a2ec11de-qv8v4"]
Oct 30 13:22:03.148462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:03.148388    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"]
Oct 30 13:22:03.153241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:03.153214    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6b7b93a3-9pwdp"]
Oct 30 13:22:03.881010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:03.880972    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08066830-1005-43ef-9c8a-2a4bd4aa1c48" path="/var/lib/kubelet/pods/08066830-1005-43ef-9c8a-2a4bd4aa1c48/volumes"
Oct 30 13:22:03.881660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:03.881629    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6498d357-c8a3-4943-9437-3c8dae7bcea6" path="/var/lib/kubelet/pods/6498d357-c8a3-4943-9437-3c8dae7bcea6/volumes"
Oct 30 13:22:06.338261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:06.338224    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:22:08.212895 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:22:08.212857    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:22:10.416655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:10.416605    4291 scope.go:122] "RemoveContainer" containerID="dc303d86ccf66a3ac12de2ffb88ff56e092975ef309b11c3c75c6989ef72e024"
Oct 30 13:22:10.435235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:10.435170    4291 scope.go:122] "RemoveContainer" containerID="353a9089143fcad5e1a3655b9242be8387dfe3dd00154174a33329d3a5f33e91"
Oct 30 13:22:14.034074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:14.034033    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"]
Oct 30 13:22:14.045584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:14.045549    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82d5fead-pltnm"]
Oct 30 13:22:15.882591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:15.882551    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dfdb1914-f8a5-49c6-bfbb-160bb380737b" path="/var/lib/kubelet/pods/dfdb1914-f8a5-49c6-bfbb-160bb380737b/volumes"
Oct 30 13:22:22.933691 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:22.933655    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"]
Oct 30 13:22:22.981671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:22.981626    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"]
Oct 30 13:22:27.934181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:27.934101    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd" podUID="9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" containerName="payload-processing" containerID="cri-o://d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef" gracePeriod=25
Oct 30 13:22:27.982335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:27.982291    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj" podUID="43fc1425-d02d-4f14-bc76-297492f67a91" containerName="payload-pre-processing" containerID="cri-o://398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac" gracePeriod=25
Oct 30 13:22:28.267869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.267846    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:22:28.292189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.292166    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:22:28.365597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.365569    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-kube-api-access-csl6r\") pod \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") "
Oct 30 13:22:28.365803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.365681    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43fc1425-d02d-4f14-bc76-297492f67a91-kube-api-access-8ztht\" (UniqueName: \"kubernetes.io/projected/43fc1425-d02d-4f14-bc76-297492f67a91-kube-api-access-8ztht\") pod \"43fc1425-d02d-4f14-bc76-297492f67a91\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") "
Oct 30 13:22:28.365888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.365802    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-plugins-config-volume\") pod \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\" (UID: \"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95\") "
Oct 30 13:22:28.365949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.365930    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43fc1425-d02d-4f14-bc76-297492f67a91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc1425-d02d-4f14-bc76-297492f67a91-plugins-config-volume\") pod \"43fc1425-d02d-4f14-bc76-297492f67a91\" (UID: \"43fc1425-d02d-4f14-bc76-297492f67a91\") "
Oct 30 13:22:28.366402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.366368    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43fc1425-d02d-4f14-bc76-297492f67a91-plugins-config-volume" pod "43fc1425-d02d-4f14-bc76-297492f67a91" (UID: "43fc1425-d02d-4f14-bc76-297492f67a91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:28.366533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.366474    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43fc1425-d02d-4f14-bc76-297492f67a91-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:28.366533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.366515    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-plugins-config-volume" pod "9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" (UID: "9a7da5d1-93f1-47b0-824a-6f06a7b0ac95"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:28.368371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.368339    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43fc1425-d02d-4f14-bc76-297492f67a91-kube-api-access-8ztht" pod "43fc1425-d02d-4f14-bc76-297492f67a91" (UID: "43fc1425-d02d-4f14-bc76-297492f67a91"). InnerVolumeSpecName "kube-api-access-8ztht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:28.368452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.368428    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-kube-api-access-csl6r" pod "9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" (UID: "9a7da5d1-93f1-47b0-824a-6f06a7b0ac95"). InnerVolumeSpecName "kube-api-access-csl6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:28.467528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.467456    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:28.467528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.467482    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csl6r\" (UniqueName: \"kubernetes.io/projected/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95-kube-api-access-csl6r\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:28.467528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.467496    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ztht\" (UniqueName: \"kubernetes.io/projected/43fc1425-d02d-4f14-bc76-297492f67a91-kube-api-access-8ztht\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:28.601885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.601851    4291 generic.go:356] "Generic (PLEG): container finished" podID="43fc1425-d02d-4f14-bc76-297492f67a91" containerID="398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac" exitCode=0
Oct 30 13:22:28.602012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.601895    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"
Oct 30 13:22:28.602012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.601936    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj" event={"ID":"43fc1425-d02d-4f14-bc76-297492f67a91","Type":"ContainerDied","Data":"398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac"}
Oct 30 13:22:28.602114 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.602019    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj" event={"ID":"43fc1425-d02d-4f14-bc76-297492f67a91","Type":"ContainerDied","Data":"7b72813d18269e1ee4a465a914af5c8b559f7827189b23606137c217c2863520"}
Oct 30 13:22:28.602114 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.602044    4291 scope.go:122] "RemoveContainer" containerID="398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac"
Oct 30 13:22:28.603481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.603454    4291 generic.go:356] "Generic (PLEG): container finished" podID="9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" containerID="d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef" exitCode=0
Oct 30 13:22:28.603591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.603498    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"
Oct 30 13:22:28.603591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.603556    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd" event={"ID":"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95","Type":"ContainerDied","Data":"d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef"}
Oct 30 13:22:28.603728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.603606    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd" event={"ID":"9a7da5d1-93f1-47b0-824a-6f06a7b0ac95","Type":"ContainerDied","Data":"f977143b2feda3859debc9edcad28c6c7d8ab22ce2ed3d3516d54ee1517034ce"}
Oct 30 13:22:28.613588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.613565    4291 scope.go:122] "RemoveContainer" containerID="398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac"
Oct 30 13:22:28.613863 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:22:28.613844    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac\": container with ID starting with 398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac not found: ID does not exist" containerID="398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac"
Oct 30 13:22:28.613967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.613873    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac"} err="failed to get container status \"398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac\": rpc error: code = NotFound desc = could not find container \"398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac\": container with ID starting with 398f119acf8ea3b4dbe0deddff55c3b9504ba6ced8b56da009891886ff8074ac not found: ID does not exist"
Oct 30 13:22:28.613967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.613895    4291 scope.go:122] "RemoveContainer" containerID="d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef"
Oct 30 13:22:28.624131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.624111    4291 scope.go:122] "RemoveContainer" containerID="d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef"
Oct 30 13:22:28.624373 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:22:28.624357    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef\": container with ID starting with d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef not found: ID does not exist" containerID="d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef"
Oct 30 13:22:28.624423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.624379    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef"} err="failed to get container status \"d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef\": rpc error: code = NotFound desc = could not find container \"d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef\": container with ID starting with d251fe7a30b17444ddfc90bd2045a8c7ff592261c7a64ecaca1c360579ac28ef not found: ID does not exist"
Oct 30 13:22:28.654874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.654852    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"]
Oct 30 13:22:28.664702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.664680    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-56c5dc594c-lqvlj"]
Oct 30 13:22:28.687019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.686997    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"]
Oct 30 13:22:28.691979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:28.691961    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-86bb4db864-gjlnd"]
Oct 30 13:22:29.032568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:29.032533    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"]
Oct 30 13:22:29.036304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:29.036283    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3915dc57-222hm"]
Oct 30 13:22:29.878911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:29.878877    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43fc1425-d02d-4f14-bc76-297492f67a91" path="/var/lib/kubelet/pods/43fc1425-d02d-4f14-bc76-297492f67a91/volumes"
Oct 30 13:22:29.879352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:29.879337    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6627a426-e812-4ca7-8bbd-1bd2b5923ba5" path="/var/lib/kubelet/pods/6627a426-e812-4ca7-8bbd-1bd2b5923ba5/volumes"
Oct 30 13:22:29.879766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:29.879751    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" path="/var/lib/kubelet/pods/9a7da5d1-93f1-47b0-824a-6f06a7b0ac95/volumes"
Oct 30 13:22:30.038996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:30.038962    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"]
Oct 30 13:22:30.050846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:30.050814    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa228694-4f2k5"]
Oct 30 13:22:31.878887 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:31.878854    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f012eb90-91c8-4430-812a-2be3017b0b57" path="/var/lib/kubelet/pods/f012eb90-91c8-4430-812a-2be3017b0b57/volumes"
Oct 30 13:22:33.226569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:33.226527    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"]
Oct 30 13:22:33.258796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:33.258764    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"]
Oct 30 13:22:36.033910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:36.033874    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"]
Oct 30 13:22:36.038250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:36.038228    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e20ae2c5-2dgbv"]
Oct 30 13:22:37.887064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:37.887028    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46" path="/var/lib/kubelet/pods/b75cfaf0-e6a2-4d2c-b21e-c2a80ac8ab46/volumes"
Oct 30 13:22:38.212388 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:22:38.212285    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:22:38.227554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.227513    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld" podUID="2221ef3b-2cbe-49b4-a944-c80c14a051f4" containerName="payload-processing" containerID="cri-o://1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed" gracePeriod=25
Oct 30 13:22:38.259336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.259295    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4" podUID="676e64fd-59d0-4d95-819e-981d5d35d7e0" containerName="payload-pre-processing" containerID="cri-o://89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c" gracePeriod=25
Oct 30 13:22:38.582987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.582962    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:22:38.607577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.607553    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:22:38.689395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.689358    4291 generic.go:356] "Generic (PLEG): container finished" podID="676e64fd-59d0-4d95-819e-981d5d35d7e0" containerID="89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c" exitCode=0
Oct 30 13:22:38.689599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.689418    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"
Oct 30 13:22:38.689599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.689442    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4" event={"ID":"676e64fd-59d0-4d95-819e-981d5d35d7e0","Type":"ContainerDied","Data":"89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c"}
Oct 30 13:22:38.689599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.689508    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4" event={"ID":"676e64fd-59d0-4d95-819e-981d5d35d7e0","Type":"ContainerDied","Data":"256d4ade5b8abd41f414c1d583c60496cf3b4a83adac3243cbcd4f7f4332a1cd"}
Oct 30 13:22:38.689599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.689538    4291 scope.go:122] "RemoveContainer" containerID="89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c"
Oct 30 13:22:38.691399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.691375    4291 generic.go:356] "Generic (PLEG): container finished" podID="2221ef3b-2cbe-49b4-a944-c80c14a051f4" containerID="1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed" exitCode=0
Oct 30 13:22:38.691508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.691407    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld" event={"ID":"2221ef3b-2cbe-49b4-a944-c80c14a051f4","Type":"ContainerDied","Data":"1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed"}
Oct 30 13:22:38.691508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.691459    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld" event={"ID":"2221ef3b-2cbe-49b4-a944-c80c14a051f4","Type":"ContainerDied","Data":"63ea9db8b446a143aa23283cec882b8077180f1651b7b36eec9d88ccb39965ef"}
Oct 30 13:22:38.691508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.691421    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"
Oct 30 13:22:38.703198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.703178    4291 scope.go:122] "RemoveContainer" containerID="89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c"
Oct 30 13:22:38.703567 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:22:38.703540    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c\": container with ID starting with 89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c not found: ID does not exist" containerID="89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c"
Oct 30 13:22:38.703696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.703578    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c"} err="failed to get container status \"89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c\": rpc error: code = NotFound desc = could not find container \"89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c\": container with ID starting with 89602e1af938016db3b08708195f93d9b15914ebf76493eb954d720124467c1c not found: ID does not exist"
Oct 30 13:22:38.703696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.703600    4291 scope.go:122] "RemoveContainer" containerID="1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed"
Oct 30 13:22:38.712883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.712851    4291 scope.go:122] "RemoveContainer" containerID="1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed"
Oct 30 13:22:38.713128 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:22:38.713106    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed\": container with ID starting with 1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed not found: ID does not exist" containerID="1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed"
Oct 30 13:22:38.713184 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.713138    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed"} err="failed to get container status \"1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed\": rpc error: code = NotFound desc = could not find container \"1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed\": container with ID starting with 1f678c5d50477f9473688f66fb3d631f81003182c4c1c4e52564125162890eed not found: ID does not exist"
Oct 30 13:22:38.718070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718051    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2221ef3b-2cbe-49b4-a944-c80c14a051f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2221ef3b-2cbe-49b4-a944-c80c14a051f4-plugins-config-volume\") pod \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") "
Oct 30 13:22:38.718209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718191    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2221ef3b-2cbe-49b4-a944-c80c14a051f4-kube-api-access-zwmt7\" (UniqueName: \"kubernetes.io/projected/2221ef3b-2cbe-49b4-a944-c80c14a051f4-kube-api-access-zwmt7\") pod \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\" (UID: \"2221ef3b-2cbe-49b4-a944-c80c14a051f4\") "
Oct 30 13:22:38.718328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718311    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676e64fd-59d0-4d95-819e-981d5d35d7e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e64fd-59d0-4d95-819e-981d5d35d7e0-plugins-config-volume\") pod \"676e64fd-59d0-4d95-819e-981d5d35d7e0\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") "
Oct 30 13:22:38.718433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718416    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676e64fd-59d0-4d95-819e-981d5d35d7e0-kube-api-access-rrz7b\" (UniqueName: \"kubernetes.io/projected/676e64fd-59d0-4d95-819e-981d5d35d7e0-kube-api-access-rrz7b\") pod \"676e64fd-59d0-4d95-819e-981d5d35d7e0\" (UID: \"676e64fd-59d0-4d95-819e-981d5d35d7e0\") "
Oct 30 13:22:38.718474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718433    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2221ef3b-2cbe-49b4-a944-c80c14a051f4-plugins-config-volume" pod "2221ef3b-2cbe-49b4-a944-c80c14a051f4" (UID: "2221ef3b-2cbe-49b4-a944-c80c14a051f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:38.718733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718706    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676e64fd-59d0-4d95-819e-981d5d35d7e0-plugins-config-volume" pod "676e64fd-59d0-4d95-819e-981d5d35d7e0" (UID: "676e64fd-59d0-4d95-819e-981d5d35d7e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:38.718982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718967    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2221ef3b-2cbe-49b4-a944-c80c14a051f4-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:38.719047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.718986    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676e64fd-59d0-4d95-819e-981d5d35d7e0-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:38.720307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.720270    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2221ef3b-2cbe-49b4-a944-c80c14a051f4-kube-api-access-zwmt7" pod "2221ef3b-2cbe-49b4-a944-c80c14a051f4" (UID: "2221ef3b-2cbe-49b4-a944-c80c14a051f4"). InnerVolumeSpecName "kube-api-access-zwmt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:38.720396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.720347    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676e64fd-59d0-4d95-819e-981d5d35d7e0-kube-api-access-rrz7b" pod "676e64fd-59d0-4d95-819e-981d5d35d7e0" (UID: "676e64fd-59d0-4d95-819e-981d5d35d7e0"). InnerVolumeSpecName "kube-api-access-rrz7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:38.820454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.820366    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrz7b\" (UniqueName: \"kubernetes.io/projected/676e64fd-59d0-4d95-819e-981d5d35d7e0-kube-api-access-rrz7b\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:38.820454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:38.820398    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwmt7\" (UniqueName: \"kubernetes.io/projected/2221ef3b-2cbe-49b4-a944-c80c14a051f4-kube-api-access-zwmt7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:39.038805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:39.038771    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"]
Oct 30 13:22:39.042343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:39.042321    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-1f486a-568c587754-95mld"]
Oct 30 13:22:39.083174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:39.083090    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"]
Oct 30 13:22:39.086528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:39.086506    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-1f486a-6b77ddfc6c-x2gk4"]
Oct 30 13:22:39.880447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:39.880409    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2221ef3b-2cbe-49b4-a944-c80c14a051f4" path="/var/lib/kubelet/pods/2221ef3b-2cbe-49b4-a944-c80c14a051f4/volumes"
Oct 30 13:22:39.881033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:39.881017    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676e64fd-59d0-4d95-819e-981d5d35d7e0" path="/var/lib/kubelet/pods/676e64fd-59d0-4d95-819e-981d5d35d7e0/volumes"
Oct 30 13:22:41.032559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:41.032521    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"]
Oct 30 13:22:41.044066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:41.044038    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2ce0e250-qd86g"]
Oct 30 13:22:41.885629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:41.885581    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0472bdc0-8c4c-4970-9800-dd95c5b9feb6" path="/var/lib/kubelet/pods/0472bdc0-8c4c-4970-9800-dd95c5b9feb6/volumes"
Oct 30 13:22:43.059542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.059508    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:22:43.600179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600139    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"]
Oct 30 13:22:43.600845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600822    4291 cpu_manager.go:414] "containerMap: removing container" podUID="676e64fd-59d0-4d95-819e-981d5d35d7e0" containerName="payload-pre-processing"
Oct 30 13:22:43.600845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600845    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676e64fd-59d0-4d95-819e-981d5d35d7e0" containerName="payload-pre-processing"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600893    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" containerName="payload-processing"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600903    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" containerName="payload-processing"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600921    4291 cpu_manager.go:414] "containerMap: removing container" podUID="08066830-1005-43ef-9c8a-2a4bd4aa1c48" containerName="revoke-keys"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600930    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08066830-1005-43ef-9c8a-2a4bd4aa1c48" containerName="revoke-keys"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600950    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0e125e83-56b5-4168-a1a5-40bae1600fad" containerName="revoke-keys"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600957    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e125e83-56b5-4168-a1a5-40bae1600fad" containerName="revoke-keys"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600973    4291 cpu_manager.go:414] "containerMap: removing container" podUID="43fc1425-d02d-4f14-bc76-297492f67a91" containerName="payload-pre-processing"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.600985    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43fc1425-d02d-4f14-bc76-297492f67a91" containerName="payload-pre-processing"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601004    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2221ef3b-2cbe-49b4-a944-c80c14a051f4" containerName="payload-processing"
Oct 30 13:22:43.601016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601013    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2221ef3b-2cbe-49b4-a944-c80c14a051f4" containerName="payload-processing"
Oct 30 13:22:43.601367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601217    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a7da5d1-93f1-47b0-824a-6f06a7b0ac95" containerName="payload-processing"
Oct 30 13:22:43.601367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601230    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="08066830-1005-43ef-9c8a-2a4bd4aa1c48" containerName="revoke-keys"
Oct 30 13:22:43.601367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601240    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e125e83-56b5-4168-a1a5-40bae1600fad" containerName="revoke-keys"
Oct 30 13:22:43.601367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601248    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="43fc1425-d02d-4f14-bc76-297492f67a91" containerName="payload-pre-processing"
Oct 30 13:22:43.601367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601258    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2221ef3b-2cbe-49b4-a944-c80c14a051f4" containerName="payload-processing"
Oct 30 13:22:43.601367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.601271    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="676e64fd-59d0-4d95-819e-981d5d35d7e0" containerName="payload-pre-processing"
Oct 30 13:22:43.606675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.606625    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:43.624796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.623323    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"]
Oct 30 13:22:43.625710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.625690    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-5dcf84\""
Oct 30 13:22:43.633519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.630581    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-5dcf84-dockercfg-d2kbb\""
Oct 30 13:22:43.783823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.781029    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"]
Oct 30 13:22:43.786165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.786136    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:43.793779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.793734    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48935990-b8d4-4d88-a661-0eb0a581e2e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:43.795957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.795928    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gst48\" (UniqueName: \"kubernetes.io/projected/48935990-b8d4-4d88-a661-0eb0a581e2e1-kube-api-access-gst48\") pod \"payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:43.805653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.803312    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"]
Oct 30 13:22:43.912709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.903490    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48935990-b8d4-4d88-a661-0eb0a581e2e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:43.912709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.903546    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d56w\" (UniqueName: \"kubernetes.io/projected/ff8dc794-a677-4869-b002-6a1679093f57-kube-api-access-6d56w\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:43.912709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.903629    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8dc794-a677-4869-b002-6a1679093f57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:43.912709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.903805    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gst48\" (UniqueName: \"kubernetes.io/projected/48935990-b8d4-4d88-a661-0eb0a581e2e1-kube-api-access-gst48\") pod \"payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:43.912709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.904933    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48935990-b8d4-4d88-a661-0eb0a581e2e1-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:43.937659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:43.936619    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gst48\" (UniqueName: \"kubernetes.io/projected/48935990-b8d4-4d88-a661-0eb0a581e2e1-kube-api-access-gst48\") pod \"payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:44.006699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.005368    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d56w\" (UniqueName: \"kubernetes.io/projected/ff8dc794-a677-4869-b002-6a1679093f57-kube-api-access-6d56w\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:44.006699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.005502    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8dc794-a677-4869-b002-6a1679093f57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:44.006920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.006903    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8dc794-a677-4869-b002-6a1679093f57-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:44.029361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.029329    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d56w\" (UniqueName: \"kubernetes.io/projected/ff8dc794-a677-4869-b002-6a1679093f57-kube-api-access-6d56w\") pod \"payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:44.113195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.113165    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:44.231706 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.226830    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:44.688396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.687163    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"]
Oct 30 13:22:44.766994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.766006    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"]
Oct 30 13:22:44.823689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:44.801036    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k" event={"ID":"ff8dc794-a677-4869-b002-6a1679093f57","Type":"ContainerStarted","Data":"34268aa7baf4e351e50c44c38b35230ec5c4558fa84c42cbeab2b34dcb0163cf"}
Oct 30 13:22:45.697662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.697197    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"]
Oct 30 13:22:45.739050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.738468    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-3c2202-l4frb"]
Oct 30 13:22:45.846145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.845945    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf" event={"ID":"48935990-b8d4-4d88-a661-0eb0a581e2e1","Type":"ContainerStarted","Data":"455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee"}
Oct 30 13:22:45.846145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.846015    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf" event={"ID":"48935990-b8d4-4d88-a661-0eb0a581e2e1","Type":"ContainerStarted","Data":"55b37d04706e3804995e2bed99946fd2275382d2d3a5d1564a010d77a3de1e3c"}
Oct 30 13:22:45.847139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.847113    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:45.868656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.854042    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k" event={"ID":"ff8dc794-a677-4869-b002-6a1679093f57","Type":"ContainerStarted","Data":"36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf"}
Oct 30 13:22:45.868656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.854861    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:45.897656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.896858    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf" podStartSLOduration=2.896838731 podStartE2EDuration="2.896838731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:45.894923891 +0000 UTC m=+1176.155162130" watchObservedRunningTime="2026-10-30 13:22:45.896838731 +0000 UTC m=+1176.157076968"
Oct 30 13:22:45.967767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:45.967413    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k" podStartSLOduration=2.967387817 podStartE2EDuration="2.967387817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:45.948857842 +0000 UTC m=+1176.209096079" watchObservedRunningTime="2026-10-30 13:22:45.967387817 +0000 UTC m=+1176.227626052"
Oct 30 13:22:46.010266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:46.008929    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e125e83-56b5-4168-a1a5-40bae1600fad" path="/var/lib/kubelet/pods/0e125e83-56b5-4168-a1a5-40bae1600fad/volumes"
Oct 30 13:22:50.036601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.036566    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"]
Oct 30 13:22:50.048436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.048403    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"]
Oct 30 13:22:50.082201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.082157    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e699e426-r4nlf"]
Oct 30 13:22:50.088069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.088030    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eac47664-72zjf"]
Oct 30 13:22:50.179764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.179727    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"]
Oct 30 13:22:50.185153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.185128    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.199241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.199215    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-8fb1a8\""
Oct 30 13:22:50.202130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.202099    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"]
Oct 30 13:22:50.235671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.235620    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"]
Oct 30 13:22:50.240010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.239983    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.242332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.242307    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-8fb1a8-dockercfg-x4rlz\""
Oct 30 13:22:50.267593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.267560    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"]
Oct 30 13:22:50.317722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.313002    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a766dbd-6979-47b9-b75c-2385a43066c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.317722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.313712    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z64vk\" (UniqueName: \"kubernetes.io/projected/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-kube-api-access-z64vk\") pod \"payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.317722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.313979    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ck95k\" (UniqueName: \"kubernetes.io/projected/7a766dbd-6979-47b9-b75c-2385a43066c9-kube-api-access-ck95k\") pod \"payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.317722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.314064    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.430566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.427556    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.430566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.427701    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a766dbd-6979-47b9-b75c-2385a43066c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.430566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.427735    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z64vk\" (UniqueName: \"kubernetes.io/projected/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-kube-api-access-z64vk\") pod \"payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.430566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.427930    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ck95k\" (UniqueName: \"kubernetes.io/projected/7a766dbd-6979-47b9-b75c-2385a43066c9-kube-api-access-ck95k\") pod \"payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.430566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.430345    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.467526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.467493    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a766dbd-6979-47b9-b75c-2385a43066c9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.487410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.487289    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ck95k\" (UniqueName: \"kubernetes.io/projected/7a766dbd-6979-47b9-b75c-2385a43066c9-kube-api-access-ck95k\") pod \"payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") " pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.491852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.491810    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z64vk\" (UniqueName: \"kubernetes.io/projected/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-kube-api-access-z64vk\") pod \"payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.501134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.501114    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:50.566449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.566419    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:50.835970 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.835940    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"]
Oct 30 13:22:50.836780 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:22:50.836754    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a766dbd_6979_47b9_b75c_2385a43066c9.slice/crio-fe84146ff67e7c9be40fa40f7e8a8d32f4885e786a20e7cc1130771d558f241c WatchSource:0}: Error finding container fe84146ff67e7c9be40fa40f7e8a8d32f4885e786a20e7cc1130771d558f241c: Status 404 returned error can't find the container with id fe84146ff67e7c9be40fa40f7e8a8d32f4885e786a20e7cc1130771d558f241c
Oct 30 13:22:50.906230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.906201    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"]
Oct 30 13:22:50.911044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:50.911018    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk" event={"ID":"7a766dbd-6979-47b9-b75c-2385a43066c9","Type":"ContainerStarted","Data":"fe84146ff67e7c9be40fa40f7e8a8d32f4885e786a20e7cc1130771d558f241c"}
Oct 30 13:22:50.913160 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:22:50.913135    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca49c6f6_faa0_4a22_ad62_a552ec4484d5.slice/crio-5aad12bf4dcd76364efa621d11af51c70b2166650122b7a51fcb83638fa18668 WatchSource:0}: Error finding container 5aad12bf4dcd76364efa621d11af51c70b2166650122b7a51fcb83638fa18668: Status 404 returned error can't find the container with id 5aad12bf4dcd76364efa621d11af51c70b2166650122b7a51fcb83638fa18668
Oct 30 13:22:51.881180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.881137    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c2e2347-a7af-46e0-a450-94b9bdf15ed5" path="/var/lib/kubelet/pods/2c2e2347-a7af-46e0-a450-94b9bdf15ed5/volumes"
Oct 30 13:22:51.881799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.881781    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="625617c8-abd6-4512-8da7-13847dc2df39" path="/var/lib/kubelet/pods/625617c8-abd6-4512-8da7-13847dc2df39/volumes"
Oct 30 13:22:51.925567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.925529    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk" event={"ID":"7a766dbd-6979-47b9-b75c-2385a43066c9","Type":"ContainerStarted","Data":"ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc"}
Oct 30 13:22:51.925785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.925621    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:22:51.927630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.927603    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr" event={"ID":"ca49c6f6-faa0-4a22-ad62-a552ec4484d5","Type":"ContainerStarted","Data":"4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241"}
Oct 30 13:22:51.927797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.927666    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr" event={"ID":"ca49c6f6-faa0-4a22-ad62-a552ec4484d5","Type":"ContainerStarted","Data":"5aad12bf4dcd76364efa621d11af51c70b2166650122b7a51fcb83638fa18668"}
Oct 30 13:22:51.927797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.927704    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:22:51.969223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:51.969149    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk" podStartSLOduration=1.9691297269999999 podStartE2EDuration="1.969129727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:51.965967516 +0000 UTC m=+1182.226205752" watchObservedRunningTime="2026-10-30 13:22:51.969129727 +0000 UTC m=+1182.229367962"
Oct 30 13:22:52.004733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:52.004678    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr" podStartSLOduration=2.004663828 podStartE2EDuration="2.004663828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:22:52.003581558 +0000 UTC m=+1182.263819793" watchObservedRunningTime="2026-10-30 13:22:52.004663828 +0000 UTC m=+1182.264902054"
Oct 30 13:22:53.823380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.823345    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"]
Oct 30 13:22:53.828848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.828825    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.845203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.845177    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"]
Oct 30 13:22:53.851798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.851768    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:22:53.851798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.851768    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:22:53.854683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.854626    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:22:53.854683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.854678    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:22:53.888745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.888692    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.888953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.888803    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxtgs\" (UniqueName: \"kubernetes.io/projected/2d3a0038-b096-4f6f-aa2e-5f22d286851d-kube-api-access-xxtgs\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.888953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.888877    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.990128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.990091    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.991407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.990282    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxtgs\" (UniqueName: \"kubernetes.io/projected/2d3a0038-b096-4f6f-aa2e-5f22d286851d-kube-api-access-xxtgs\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.991407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.990360    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.991407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.990956    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:53.992483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:53.992462    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:54.009670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.009625    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxtgs\" (UniqueName: \"kubernetes.io/projected/2d3a0038-b096-4f6f-aa2e-5f22d286851d-kube-api-access-xxtgs\") pod \"maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:54.036711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.036681    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"]
Oct 30 13:22:54.040187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.040159    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf80ac2c-d8kpf"]
Oct 30 13:22:54.146800 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.146773    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:54.454317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.454291    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"]
Oct 30 13:22:54.456055 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:22:54.456030    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3a0038_b096_4f6f_aa2e_5f22d286851d.slice/crio-83af675e062cb07f93612b4b44794c78e15e16c4aedaaebd13ccd57db85401c2 WatchSource:0}: Error finding container 83af675e062cb07f93612b4b44794c78e15e16c4aedaaebd13ccd57db85401c2: Status 404 returned error can't find the container with id 83af675e062cb07f93612b4b44794c78e15e16c4aedaaebd13ccd57db85401c2
Oct 30 13:22:54.952226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.952192    4291 generic.go:356] "Generic (PLEG): container finished" podID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerID="ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34" exitCode=0
Oct 30 13:22:54.952625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.952260    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" event={"ID":"533c0e6f-0446-40d9-b4fa-58fbd1c34c59","Type":"ContainerDied","Data":"ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34"}
Oct 30 13:22:54.952735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.952717    4291 scope.go:122] "RemoveContainer" containerID="ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34"
Oct 30 13:22:54.954017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.953991    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb" event={"ID":"2d3a0038-b096-4f6f-aa2e-5f22d286851d","Type":"ContainerStarted","Data":"83af675e062cb07f93612b4b44794c78e15e16c4aedaaebd13ccd57db85401c2"}
Oct 30 13:22:54.954321 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.954294    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:54.954376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.954363    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:54.954412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:54.954402    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:22:55.890797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:55.890754    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2a45093-dc1e-41f9-b7b1-b3d009c75d8c" path="/var/lib/kubelet/pods/a2a45093-dc1e-41f9-b7b1-b3d009c75d8c/volumes"
Oct 30 13:22:56.240808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:56.240334    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" event={"ID":"533c0e6f-0446-40d9-b4fa-58fbd1c34c59","Type":"ContainerStarted","Data":"fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410"}
Oct 30 13:22:56.246556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:56.246510    4291 generic.go:356] "Generic (PLEG): container finished" podID="2d3a0038-b096-4f6f-aa2e-5f22d286851d" containerID="95ff3d3d02135e3f2e4db279cb0108e70b2392484c785add45f54895147ae5d1" exitCode=0
Oct 30 13:22:56.247425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:56.246880    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb" event={"ID":"2d3a0038-b096-4f6f-aa2e-5f22d286851d","Type":"ContainerDied","Data":"95ff3d3d02135e3f2e4db279cb0108e70b2392484c785add45f54895147ae5d1"}
Oct 30 13:22:56.794046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:56.794013    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:22:56.794245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:56.794227    4291 patch_prober.go:29] interesting pod/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:22:56.794310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:56.794278    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 13:22:57.491982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.491950    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:57.587801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.586470    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d3a0038-b096-4f6f-aa2e-5f22d286851d-kube-api-access-xxtgs\" (UniqueName: \"kubernetes.io/projected/2d3a0038-b096-4f6f-aa2e-5f22d286851d-kube-api-access-xxtgs\") pod \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") "
Oct 30 13:22:57.587801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.586544    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-openshift-ca\") pod \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") "
Oct 30 13:22:57.587801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.586581    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-xks-ca\") pod \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\" (UID: \"2d3a0038-b096-4f6f-aa2e-5f22d286851d\") "
Oct 30 13:22:57.588572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.588539    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-openshift-ca" pod "2d3a0038-b096-4f6f-aa2e-5f22d286851d" (UID: "2d3a0038-b096-4f6f-aa2e-5f22d286851d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:22:57.595166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.595090    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-xks-ca" pod "2d3a0038-b096-4f6f-aa2e-5f22d286851d" (UID: "2d3a0038-b096-4f6f-aa2e-5f22d286851d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:22:57.595773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.595743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d3a0038-b096-4f6f-aa2e-5f22d286851d-kube-api-access-xxtgs" pod "2d3a0038-b096-4f6f-aa2e-5f22d286851d" (UID: "2d3a0038-b096-4f6f-aa2e-5f22d286851d"). InnerVolumeSpecName "kube-api-access-xxtgs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:22:57.688972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.688927    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxtgs\" (UniqueName: \"kubernetes.io/projected/2d3a0038-b096-4f6f-aa2e-5f22d286851d-kube-api-access-xxtgs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:57.688972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.688961    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:57.688972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.688977    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2d3a0038-b096-4f6f-aa2e-5f22d286851d-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:22:57.794429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.794389    4291 patch_prober.go:29] interesting pod/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:22:57.794613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.794455    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 13:22:57.895096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.895063    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:22:57.895294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:57.895216    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:22:58.283305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:58.283221    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"
Oct 30 13:22:58.283305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:58.283245    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb" event={"ID":"2d3a0038-b096-4f6f-aa2e-5f22d286851d","Type":"ContainerDied","Data":"83af675e062cb07f93612b4b44794c78e15e16c4aedaaebd13ccd57db85401c2"}
Oct 30 13:22:58.283507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:58.283322    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83af675e062cb07f93612b4b44794c78e15e16c4aedaaebd13ccd57db85401c2"
Oct 30 13:22:58.793458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:58.793421    4291 patch_prober.go:29] interesting pod/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:22:58.793863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:58.793474    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 13:22:59.793549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:59.793516    4291 patch_prober.go:29] interesting pod/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:22:59.793968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:22:59.793566    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 13:23:00.800661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:00.798247    4291 patch_prober.go:29] interesting pod/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused" start-of-body=
Oct 30 13:23:00.800661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:00.798311    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.108:15021/healthz/ready\": dial tcp 10.133.0.108:15021: connect: connection refused"
Oct 30 13:23:01.917914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:01.917880    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:23:01.931659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:01.920097    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:23:01.932036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:01.932013    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:23:02.939922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:02.939891    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:23:02.940301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:02.939961    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:23:04.036348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:04.036314    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"]
Oct 30 13:23:04.048069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:04.048035    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bd0ad0d-xmw2f"]
Oct 30 13:23:05.032060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:05.032027    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"]
Oct 30 13:23:05.044997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:05.044957    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ebda273e-hgl7k"]
Oct 30 13:23:05.879308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:05.879277    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37fc1856-c68c-4da6-b6b4-928f13d1fdbe" path="/var/lib/kubelet/pods/37fc1856-c68c-4da6-b6b4-928f13d1fdbe/volumes"
Oct 30 13:23:05.879786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:05.879771    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a0a6daa-b398-4973-8c67-ac04fa3574e0" path="/var/lib/kubelet/pods/5a0a6daa-b398-4973-8c67-ac04fa3574e0/volumes"
Oct 30 13:23:06.137611 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.137520    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"]
Oct 30 13:23:06.138291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.138269    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2d3a0038-b096-4f6f-aa2e-5f22d286851d" containerName="revoke-keys"
Oct 30 13:23:06.138398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.138292    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d3a0038-b096-4f6f-aa2e-5f22d286851d" containerName="revoke-keys"
Oct 30 13:23:06.138510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.138496    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d3a0038-b096-4f6f-aa2e-5f22d286851d" containerName="revoke-keys"
Oct 30 13:23:06.141395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.141375    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.158521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.158494    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"]
Oct 30 13:23:06.165588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.165563    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-3a82db\""
Oct 30 13:23:06.182284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.182258    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"]
Oct 30 13:23:06.185649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.185614    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.203393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.203363    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"]
Oct 30 13:23:06.223913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.223881    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-3a82db-dockercfg-js7x5\""
Oct 30 13:23:06.260062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.259998    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.260555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.260529    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.260744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.260725    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8vd\" (UniqueName: \"kubernetes.io/projected/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-kube-api-access-2j8vd\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.261052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.261024    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljlq\" (UniqueName: \"kubernetes.io/projected/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-kube-api-access-tljlq\") pod \"payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.367858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.367770    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljlq\" (UniqueName: \"kubernetes.io/projected/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-kube-api-access-tljlq\") pod \"payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.368060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.367888    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.368060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.368004    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.368060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.368049    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8vd\" (UniqueName: \"kubernetes.io/projected/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-kube-api-access-2j8vd\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.369901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.369544    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.373093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.371101    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.386926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.386901    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljlq\" (UniqueName: \"kubernetes.io/projected/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-kube-api-access-tljlq\") pod \"payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.387134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.387112    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8vd\" (UniqueName: \"kubernetes.io/projected/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-kube-api-access-2j8vd\") pod \"payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.455492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.455402    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:06.504982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.504949    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:06.766682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.766656    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"]
Oct 30 13:23:06.767409 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:23:06.767389    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff9714eb_21b0_479f_9ed1_4152ce0cefd4.slice/crio-763e65a77a7dd9d16e013884327002e676123bf57ede2af934085a7dcef45f1d WatchSource:0}: Error finding container 763e65a77a7dd9d16e013884327002e676123bf57ede2af934085a7dcef45f1d: Status 404 returned error can't find the container with id 763e65a77a7dd9d16e013884327002e676123bf57ede2af934085a7dcef45f1d
Oct 30 13:23:06.841739 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:23:06.841705    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2915871_001e_4ec6_a2d1_a5cd3fa00e09.slice/crio-68fd348e64b82fd623a6c28eb6bd053ee6b92ca3944522f40d3e3dcee4928aea WatchSource:0}: Error finding container 68fd348e64b82fd623a6c28eb6bd053ee6b92ca3944522f40d3e3dcee4928aea: Status 404 returned error can't find the container with id 68fd348e64b82fd623a6c28eb6bd053ee6b92ca3944522f40d3e3dcee4928aea
Oct 30 13:23:06.842032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:06.842008    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"]
Oct 30 13:23:07.379320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.379280    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz" event={"ID":"f2915871-001e-4ec6-a2d1-a5cd3fa00e09","Type":"ContainerStarted","Data":"3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3"}
Oct 30 13:23:07.379851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.379514    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:07.379851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.379572    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz" event={"ID":"f2915871-001e-4ec6-a2d1-a5cd3fa00e09","Type":"ContainerStarted","Data":"68fd348e64b82fd623a6c28eb6bd053ee6b92ca3944522f40d3e3dcee4928aea"}
Oct 30 13:23:07.381072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.381039    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r" event={"ID":"ff9714eb-21b0-479f-9ed1-4152ce0cefd4","Type":"ContainerStarted","Data":"bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356"}
Oct 30 13:23:07.381195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.381084    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r" event={"ID":"ff9714eb-21b0-479f-9ed1-4152ce0cefd4","Type":"ContainerStarted","Data":"763e65a77a7dd9d16e013884327002e676123bf57ede2af934085a7dcef45f1d"}
Oct 30 13:23:07.381195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.381113    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:07.419088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.419032    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz" podStartSLOduration=1.419016263 podStartE2EDuration="1.419016263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:07.415590993 +0000 UTC m=+1197.675829239" watchObservedRunningTime="2026-10-30 13:23:07.419016263 +0000 UTC m=+1197.679254496"
Oct 30 13:23:07.459380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:07.459328    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r" podStartSLOduration=1.459314454 podStartE2EDuration="1.459314454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:07.455050495 +0000 UTC m=+1197.715288729" watchObservedRunningTime="2026-10-30 13:23:07.459314454 +0000 UTC m=+1197.719552711"
Oct 30 13:23:08.211184 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:23:08.211149    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:23:09.033953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:09.033921    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"]
Oct 30 13:23:09.045559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:09.045526    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-117e277a-kxx2k"]
Oct 30 13:23:09.881012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:09.880978    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7315958d-a84f-4082-a926-e7474762d355" path="/var/lib/kubelet/pods/7315958d-a84f-4082-a926-e7474762d355/volumes"
Oct 30 13:23:10.496578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:10.496546    4291 scope.go:122] "RemoveContainer" containerID="476652d7034ec1c27eddd46c54e0cbc880492341a27adaadaf9333c903f6090e"
Oct 30 13:23:10.506009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:10.505988    4291 scope.go:122] "RemoveContainer" containerID="1de14dfbed712a92dfe6582eb5885fde04598f5117bb8e3316411c6f24e58c0e"
Oct 30 13:23:10.514941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:10.514921    4291 scope.go:122] "RemoveContainer" containerID="84322b3bc02ad61d0717a5383fba08a143e5b257095aa2f7baa10c32533ddb62"
Oct 30 13:23:10.524098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:10.524073    4291 scope.go:122] "RemoveContainer" containerID="598d5dd1fc843e33b6bbc90ad91a93fc2e31df37c4c0c6716ce8ffe67a3696a4"
Oct 30 13:23:10.533320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:10.533298    4291 scope.go:122] "RemoveContainer" containerID="4cb7758f63b3d36d7899ac1ec5f093bb141850ee22e7c085e853afb7151b2342"
Oct 30 13:23:13.029994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:13.029956    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:23:18.032842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:18.032806    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"]
Oct 30 13:23:18.044290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:18.044257    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-27de93c7-5299z"]
Oct 30 13:23:18.391782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:18.391752    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:23:18.391953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:18.391822    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:23:19.881300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:19.881258    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79e878c2-d9c0-4434-addc-04fbc0894c89" path="/var/lib/kubelet/pods/79e878c2-d9c0-4434-addc-04fbc0894c89/volumes"
Oct 30 13:23:29.035529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:29.035449    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"]
Oct 30 13:23:29.046921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:29.046887    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e03fc693-nq6wg"]
Oct 30 13:23:29.883445 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:29.883404    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5" path="/var/lib/kubelet/pods/0afc7bf6-8c6a-4f09-9f22-7d629cfe32f5/volumes"
Oct 30 13:23:38.211655 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:23:38.211602    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:23:40.377023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.376986    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"]
Oct 30 13:23:40.381132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.381109    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.398449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.398412    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"]
Oct 30 13:23:40.405271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.405237    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:23:40.406871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.406839    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:23:40.407055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.406875    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:23:40.407055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.406882    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:23:40.496862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.496827    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkdj6\" (UniqueName: \"kubernetes.io/projected/c12b5bbd-8843-40bc-b0b9-8635951f49bc-kube-api-access-fkdj6\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.497048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.496979    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.497048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.497024    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.598383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.598345    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkdj6\" (UniqueName: \"kubernetes.io/projected/c12b5bbd-8843-40bc-b0b9-8635951f49bc-kube-api-access-fkdj6\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.598575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.598448    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.598575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.598489    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.599268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.599236    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.600941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.600920    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.617868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.617832    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkdj6\" (UniqueName: \"kubernetes.io/projected/c12b5bbd-8843-40bc-b0b9-8635951f49bc-kube-api-access-fkdj6\") pod \"maas-api-revoke-keys-e2e-api-a-121d75-t2k5d\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:40.695322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:40.695235    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:41.000111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:41.000078    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"]
Oct 30 13:23:41.001323 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:23:41.001296    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc12b5bbd_8843_40bc_b0b9_8635951f49bc.slice/crio-189fb474b205d2563a5c0c1bc9e0c677e3e88a652fd159d608efe2870941d29d WatchSource:0}: Error finding container 189fb474b205d2563a5c0c1bc9e0c677e3e88a652fd159d608efe2870941d29d: Status 404 returned error can't find the container with id 189fb474b205d2563a5c0c1bc9e0c677e3e88a652fd159d608efe2870941d29d
Oct 30 13:23:41.663013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:41.662921    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d" event={"ID":"c12b5bbd-8843-40bc-b0b9-8635951f49bc","Type":"ContainerStarted","Data":"0c5c39f4702f8649c16822165292aa05f2904f5f2fd16158dab72a10235b6c4b"}
Oct 30 13:23:41.663013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:41.662976    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d" event={"ID":"c12b5bbd-8843-40bc-b0b9-8635951f49bc","Type":"ContainerStarted","Data":"189fb474b205d2563a5c0c1bc9e0c677e3e88a652fd159d608efe2870941d29d"}
Oct 30 13:23:41.699442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:41.699388    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d" podStartSLOduration=1.699372264 podStartE2EDuration="1.699372264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:41.699242413 +0000 UTC m=+1231.959480646" watchObservedRunningTime="2026-10-30 13:23:41.699372264 +0000 UTC m=+1231.959610498"
Oct 30 13:23:42.672182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:42.672145    4291 generic.go:356] "Generic (PLEG): container finished" podID="c12b5bbd-8843-40bc-b0b9-8635951f49bc" containerID="0c5c39f4702f8649c16822165292aa05f2904f5f2fd16158dab72a10235b6c4b" exitCode=0
Oct 30 13:23:42.672683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:42.672224    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d" event={"ID":"c12b5bbd-8843-40bc-b0b9-8635951f49bc","Type":"ContainerDied","Data":"0c5c39f4702f8649c16822165292aa05f2904f5f2fd16158dab72a10235b6c4b"}
Oct 30 13:23:43.859467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.859439    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:43.946919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.946833    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-openshift-ca\") pod \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") "
Oct 30 13:23:43.947060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.946992    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c12b5bbd-8843-40bc-b0b9-8635951f49bc-kube-api-access-fkdj6\" (UniqueName: \"kubernetes.io/projected/c12b5bbd-8843-40bc-b0b9-8635951f49bc-kube-api-access-fkdj6\") pod \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") "
Oct 30 13:23:43.947060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.947033    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-xks-ca\") pod \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\" (UID: \"c12b5bbd-8843-40bc-b0b9-8635951f49bc\") "
Oct 30 13:23:43.947205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.947180    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-openshift-ca" pod "c12b5bbd-8843-40bc-b0b9-8635951f49bc" (UID: "c12b5bbd-8843-40bc-b0b9-8635951f49bc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:43.947742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.947719    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:23:43.949434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.949411    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12b5bbd-8843-40bc-b0b9-8635951f49bc-kube-api-access-fkdj6" pod "c12b5bbd-8843-40bc-b0b9-8635951f49bc" (UID: "c12b5bbd-8843-40bc-b0b9-8635951f49bc"). InnerVolumeSpecName "kube-api-access-fkdj6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:43.949434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:43.949421    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-xks-ca" pod "c12b5bbd-8843-40bc-b0b9-8635951f49bc" (UID: "c12b5bbd-8843-40bc-b0b9-8635951f49bc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:23:44.048803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:44.048769    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c12b5bbd-8843-40bc-b0b9-8635951f49bc-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:23:44.048803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:44.048799    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkdj6\" (UniqueName: \"kubernetes.io/projected/c12b5bbd-8843-40bc-b0b9-8635951f49bc-kube-api-access-fkdj6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:23:44.687732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:44.687686    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d" event={"ID":"c12b5bbd-8843-40bc-b0b9-8635951f49bc","Type":"ContainerDied","Data":"189fb474b205d2563a5c0c1bc9e0c677e3e88a652fd159d608efe2870941d29d"}
Oct 30 13:23:44.687732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:44.687721    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"
Oct 30 13:23:44.687944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:44.687740    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="189fb474b205d2563a5c0c1bc9e0c677e3e88a652fd159d608efe2870941d29d"
Oct 30 13:23:46.358749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:46.358713    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"]
Oct 30 13:23:46.371361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:46.371321    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-7c0e28-k8zlb"]
Oct 30 13:23:47.879899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:47.879863    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d3a0038-b096-4f6f-aa2e-5f22d286851d" path="/var/lib/kubelet/pods/2d3a0038-b096-4f6f-aa2e-5f22d286851d/volumes"
Oct 30 13:23:50.764882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:50.764847    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"]
Oct 30 13:23:52.597233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:52.597187    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:23:53.033014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:53.032928    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"]
Oct 30 13:23:53.044678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:53.044617    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-598e0cdd-wcxd4"]
Oct 30 13:23:53.880767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:53.880735    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="305bb724-61d6-4d04-96f4-0349f3cd205c" path="/var/lib/kubelet/pods/305bb724-61d6-4d04-96f4-0349f3cd205c/volumes"
Oct 30 13:23:55.765601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:55.765554    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7" podUID="8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" containerName="payload-processing" containerID="cri-o://4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5" gracePeriod=25
Oct 30 13:23:56.149550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.149515    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:23:56.156799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.156760    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-kube-api-access-75zwt\" (UniqueName: \"kubernetes.io/projected/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-kube-api-access-75zwt\") pod \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") "
Oct 30 13:23:56.156995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.156932    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-plugins-config-volume\") pod \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\" (UID: \"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e\") "
Oct 30 13:23:56.157463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.157430    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-plugins-config-volume" pod "8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" (UID: "8b496e92-b6a4-45cc-bf94-3e7b824a0b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:23:56.157694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.157672    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:23:56.158969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.158946    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-kube-api-access-75zwt" pod "8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" (UID: "8b496e92-b6a4-45cc-bf94-3e7b824a0b4e"). InnerVolumeSpecName "kube-api-access-75zwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:23:56.258700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.258664    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75zwt\" (UniqueName: \"kubernetes.io/projected/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e-kube-api-access-75zwt\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:23:56.375146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.375064    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"]
Oct 30 13:23:56.375663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.375647    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" containerName="payload-processing"
Oct 30 13:23:56.375714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.375663    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" containerName="payload-processing"
Oct 30 13:23:56.375714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.375683    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c12b5bbd-8843-40bc-b0b9-8635951f49bc" containerName="revoke-keys"
Oct 30 13:23:56.375714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.375688    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c12b5bbd-8843-40bc-b0b9-8635951f49bc" containerName="revoke-keys"
Oct 30 13:23:56.375816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.375808    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" containerName="payload-processing"
Oct 30 13:23:56.375849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.375820    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c12b5bbd-8843-40bc-b0b9-8635951f49bc" containerName="revoke-keys"
Oct 30 13:23:56.379529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.379510    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.397513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.397473    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5cee02-openshift-default-dockercfg-b67ts\""
Oct 30 13:23:56.407341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.407289    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"]
Oct 30 13:23:56.461312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461276    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-envoy\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461435    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-certs\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461478    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-podinfo\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461495    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f2b6e2-8cf4-437f-97f8-573f66633fec-istiod-ca-cert\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461604    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-data\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461624    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-socket\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461728    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zx4p\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-kube-api-access-4zx4p\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461792    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-credential-socket\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.461844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.461831    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-token\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563072    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-data\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563123    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-socket\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563177    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zx4p\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-kube-api-access-4zx4p\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563247    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-credential-socket\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563289    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-token\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563401    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-envoy\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563561    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-certs\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563615    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-podinfo\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.563708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.563657    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f2b6e2-8cf4-437f-97f8-573f66633fec-istiod-ca-cert\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.564245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.564210    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-credential-socket\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.564548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.564522    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f2b6e2-8cf4-437f-97f8-573f66633fec-istiod-ca-cert\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.564646 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.564533    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-data\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.565292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.565259    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-socket\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.565488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.565456    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-certs\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.567223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.567196    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-envoy\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.568096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.568065    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-podinfo\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.610657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.602703    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zx4p\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-kube-api-access-4zx4p\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.614659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.611203    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-token\") pod \"e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") " pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.706185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.706078    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:56.832821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.832783    4291 generic.go:356] "Generic (PLEG): container finished" podID="8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" containerID="4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5" exitCode=0
Oct 30 13:23:56.835334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.834043    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"
Oct 30 13:23:56.835603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.835561    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7" event={"ID":"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e","Type":"ContainerDied","Data":"4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5"}
Oct 30 13:23:56.835810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.835779    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7" event={"ID":"8b496e92-b6a4-45cc-bf94-3e7b824a0b4e","Type":"ContainerDied","Data":"76d7b322245fe79b445a7e9dcc15179b14eb4b602d85dc8ed7820129ec93cfce"}
Oct 30 13:23:56.836399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.835910    4291 scope.go:122] "RemoveContainer" containerID="4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5"
Oct 30 13:23:56.855062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.855018    4291 scope.go:122] "RemoveContainer" containerID="4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5"
Oct 30 13:23:56.855457 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:23:56.855431    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5\": container with ID starting with 4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5 not found: ID does not exist" containerID="4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5"
Oct 30 13:23:56.855549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.855470    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5"} err="failed to get container status \"4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5\": rpc error: code = NotFound desc = could not find container \"4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5\": container with ID starting with 4a927af3a0f4683bd90b88d0c92a429de4f5a9af7fda7cf4d204f5a91c901fd5 not found: ID does not exist"
Oct 30 13:23:56.895262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.895223    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"]
Oct 30 13:23:56.900987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:56.900890    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-121d75-5b5f9f7f58-f5lz7"]
Oct 30 13:23:57.076071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:57.076032    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"]
Oct 30 13:23:57.078040 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:23:57.078001    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50f2b6e2_8cf4_437f_97f8_573f66633fec.slice/crio-5192332f31f936143956921e472a713dc0339f9e43ab9718b5c73e30a97d17c2 WatchSource:0}: Error finding container 5192332f31f936143956921e472a713dc0339f9e43ab9718b5c73e30a97d17c2: Status 404 returned error can't find the container with id 5192332f31f936143956921e472a713dc0339f9e43ab9718b5c73e30a97d17c2
Oct 30 13:23:57.080957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:57.080914    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:57.081054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:57.081002    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:57.081095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:57.081051    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:23:57.843264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:57.843221    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" event={"ID":"50f2b6e2-8cf4-437f-97f8-573f66633fec","Type":"ContainerStarted","Data":"23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab"}
Oct 30 13:23:57.843719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:57.843286    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" event={"ID":"50f2b6e2-8cf4-437f-97f8-573f66633fec","Type":"ContainerStarted","Data":"5192332f31f936143956921e472a713dc0339f9e43ab9718b5c73e30a97d17c2"}
Oct 30 13:23:57.883652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:57.883530    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b496e92-b6a4-45cc-bf94-3e7b824a0b4e" path="/var/lib/kubelet/pods/8b496e92-b6a4-45cc-bf94-3e7b824a0b4e/volumes"
Oct 30 13:23:58.049431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.049373    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" podStartSLOduration=2.049354896 podStartE2EDuration="2.049354896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:57.893489741 +0000 UTC m=+1248.153727975" watchObservedRunningTime="2026-10-30 13:23:58.049354896 +0000 UTC m=+1248.309593137"
Oct 30 13:23:58.050980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.050955    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"]
Oct 30 13:23:58.055021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.054990    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.072924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.072889    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"]
Oct 30 13:23:58.073125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.073110    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e3e617\""
Oct 30 13:23:58.097248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.097152    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e3e617-dockercfg-2xbx7\""
Oct 30 13:23:58.097551 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.097522    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7d27a-af57-46ef-be85-409ed651042c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.097991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.097914    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlbk4\" (UniqueName: \"kubernetes.io/projected/86d7d27a-af57-46ef-be85-409ed651042c-kube-api-access-tlbk4\") pod \"payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.110181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.110137    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"]
Oct 30 13:23:58.115434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.115406    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.132117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.132086    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"]
Oct 30 13:23:58.257087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.257047    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlbk4\" (UniqueName: \"kubernetes.io/projected/86d7d27a-af57-46ef-be85-409ed651042c-kube-api-access-tlbk4\") pod \"payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.257468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.257439    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cfe0b4-bc28-4e95-966d-840c3167da56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.257862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.257840    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7d27a-af57-46ef-be85-409ed651042c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.258111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.258091    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw29q\" (UniqueName: \"kubernetes.io/projected/53cfe0b4-bc28-4e95-966d-840c3167da56-kube-api-access-gw29q\") pod \"payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.259727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.259654    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7d27a-af57-46ef-be85-409ed651042c-plugins-config-volume\") pod \"payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.303838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.303791    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlbk4\" (UniqueName: \"kubernetes.io/projected/86d7d27a-af57-46ef-be85-409ed651042c-kube-api-access-tlbk4\") pod \"payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") " pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.369838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.369743    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw29q\" (UniqueName: \"kubernetes.io/projected/53cfe0b4-bc28-4e95-966d-840c3167da56-kube-api-access-gw29q\") pod \"payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.370389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.370358    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cfe0b4-bc28-4e95-966d-840c3167da56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.371738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.371718    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cfe0b4-bc28-4e95-966d-840c3167da56-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.372323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.372304    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:58.408457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.408426    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw29q\" (UniqueName: \"kubernetes.io/projected/53cfe0b4-bc28-4e95-966d-840c3167da56-kube-api-access-gw29q\") pod \"payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.462566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.462529    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:58.706781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.706700    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:58.712812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.712785    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:58.740902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.740874    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"]
Oct 30 13:23:58.741903 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:23:58.741877    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86d7d27a_af57_46ef_be85_409ed651042c.slice/crio-8ba4f5e30d2bdc755ee29abe1a8ab767150cf6a437cb4ee6c319b7d9316e9ce9 WatchSource:0}: Error finding container 8ba4f5e30d2bdc755ee29abe1a8ab767150cf6a437cb4ee6c319b7d9316e9ce9: Status 404 returned error can't find the container with id 8ba4f5e30d2bdc755ee29abe1a8ab767150cf6a437cb4ee6c319b7d9316e9ce9
Oct 30 13:23:58.801430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.801391    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"]
Oct 30 13:23:58.803528 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:23:58.803497    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cfe0b4_bc28_4e95_966d_840c3167da56.slice/crio-9170e7bee0ac9864ab35efcb990de968897b62abd2dcf24a324814293f36d552 WatchSource:0}: Error finding container 9170e7bee0ac9864ab35efcb990de968897b62abd2dcf24a324814293f36d552: Status 404 returned error can't find the container with id 9170e7bee0ac9864ab35efcb990de968897b62abd2dcf24a324814293f36d552
Oct 30 13:23:58.856484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.856448    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p" event={"ID":"53cfe0b4-bc28-4e95-966d-840c3167da56","Type":"ContainerStarted","Data":"9170e7bee0ac9864ab35efcb990de968897b62abd2dcf24a324814293f36d552"}
Oct 30 13:23:58.858863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.858837    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr" event={"ID":"86d7d27a-af57-46ef-be85-409ed651042c","Type":"ContainerStarted","Data":"a427493084ea16af6c0f5c09f95f0892b5d4621a6d26081d25a7689cc1c1141f"}
Oct 30 13:23:58.859159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.858885    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr" event={"ID":"86d7d27a-af57-46ef-be85-409ed651042c","Type":"ContainerStarted","Data":"8ba4f5e30d2bdc755ee29abe1a8ab767150cf6a437cb4ee6c319b7d9316e9ce9"}
Oct 30 13:23:58.859159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.859038    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:58.860168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.860140    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:23:58.901509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:58.901456    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr" podStartSLOduration=0.901440692 podStartE2EDuration="901.440692ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:58.898764556 +0000 UTC m=+1249.159002802" watchObservedRunningTime="2026-10-30 13:23:58.901440692 +0000 UTC m=+1249.161678925"
Oct 30 13:23:59.868441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:59.868400    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p" event={"ID":"53cfe0b4-bc28-4e95-966d-840c3167da56","Type":"ContainerStarted","Data":"49b24f8d6fe42e696322e53d0d4804c9953d93efe2697bb0fad250e878650997"}
Oct 30 13:23:59.868986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:59.868607    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:23:59.868986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:59.868654    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:23:59.908323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:23:59.908249    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p" podStartSLOduration=1.908232401 podStartE2EDuration="1.908232401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:23:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:23:59.905403449 +0000 UTC m=+1250.165641683" watchObservedRunningTime="2026-10-30 13:23:59.908232401 +0000 UTC m=+1250.168470637"
Oct 30 13:24:00.401857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:00.401819    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"]
Oct 30 13:24:00.443013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:00.442979    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"]
Oct 30 13:24:03.032224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:03.032185    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"]
Oct 30 13:24:03.043583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:03.043552    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d47bd17-tpngt"]
Oct 30 13:24:03.880757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:03.880723    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52d4af64-7a42-4ba0-a231-466918618781" path="/var/lib/kubelet/pods/52d4af64-7a42-4ba0-a231-466918618781/volumes"
Oct 30 13:24:05.402647 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.402565    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r" podUID="ff9714eb-21b0-479f-9ed1-4152ce0cefd4" containerName="payload-processing" containerID="cri-o://bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356" gracePeriod=25
Oct 30 13:24:05.443497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.443426    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz" podUID="f2915871-001e-4ec6-a2d1-a5cd3fa00e09" containerName="payload-pre-processing" containerID="cri-o://3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3" gracePeriod=25
Oct 30 13:24:05.728715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.728240    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:24:05.761556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.761530    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:24:05.826041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.826004    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-kube-api-access-2j8vd\" (UniqueName: \"kubernetes.io/projected/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-kube-api-access-2j8vd\") pod \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") "
Oct 30 13:24:05.826241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.826131    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-plugins-config-volume\") pod \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\" (UID: \"f2915871-001e-4ec6-a2d1-a5cd3fa00e09\") "
Oct 30 13:24:05.826241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.826183    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-plugins-config-volume\") pod \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") "
Oct 30 13:24:05.826401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.826256    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-kube-api-access-tljlq\" (UniqueName: \"kubernetes.io/projected/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-kube-api-access-tljlq\") pod \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\" (UID: \"ff9714eb-21b0-479f-9ed1-4152ce0cefd4\") "
Oct 30 13:24:05.826603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.826575    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-plugins-config-volume" pod "f2915871-001e-4ec6-a2d1-a5cd3fa00e09" (UID: "f2915871-001e-4ec6-a2d1-a5cd3fa00e09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:05.827089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.827059    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-plugins-config-volume" pod "ff9714eb-21b0-479f-9ed1-4152ce0cefd4" (UID: "ff9714eb-21b0-479f-9ed1-4152ce0cefd4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:05.827235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.827210    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.828525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.828497    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-kube-api-access-tljlq" pod "ff9714eb-21b0-479f-9ed1-4152ce0cefd4" (UID: "ff9714eb-21b0-479f-9ed1-4152ce0cefd4"). InnerVolumeSpecName "kube-api-access-tljlq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.828892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.828871    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-kube-api-access-2j8vd" pod "f2915871-001e-4ec6-a2d1-a5cd3fa00e09" (UID: "f2915871-001e-4ec6-a2d1-a5cd3fa00e09"). InnerVolumeSpecName "kube-api-access-2j8vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:05.929117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.929072    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.929117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.929113    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljlq\" (UniqueName: \"kubernetes.io/projected/ff9714eb-21b0-479f-9ed1-4152ce0cefd4-kube-api-access-tljlq\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.929363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.929129    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8vd\" (UniqueName: \"kubernetes.io/projected/f2915871-001e-4ec6-a2d1-a5cd3fa00e09-kube-api-access-2j8vd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:05.944354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.944322    4291 generic.go:356] "Generic (PLEG): container finished" podID="f2915871-001e-4ec6-a2d1-a5cd3fa00e09" containerID="3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3" exitCode=0
Oct 30 13:24:05.944530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.944379    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"
Oct 30 13:24:05.944530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.944412    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz" event={"ID":"f2915871-001e-4ec6-a2d1-a5cd3fa00e09","Type":"ContainerDied","Data":"3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3"}
Oct 30 13:24:05.944530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.944494    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz" event={"ID":"f2915871-001e-4ec6-a2d1-a5cd3fa00e09","Type":"ContainerDied","Data":"68fd348e64b82fd623a6c28eb6bd053ee6b92ca3944522f40d3e3dcee4928aea"}
Oct 30 13:24:05.944666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.944528    4291 scope.go:122] "RemoveContainer" containerID="3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3"
Oct 30 13:24:05.946000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.945981    4291 generic.go:356] "Generic (PLEG): container finished" podID="ff9714eb-21b0-479f-9ed1-4152ce0cefd4" containerID="bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356" exitCode=0
Oct 30 13:24:05.946117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.946032    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"
Oct 30 13:24:05.946117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.946061    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r" event={"ID":"ff9714eb-21b0-479f-9ed1-4152ce0cefd4","Type":"ContainerDied","Data":"bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356"}
Oct 30 13:24:05.946117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.946109    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r" event={"ID":"ff9714eb-21b0-479f-9ed1-4152ce0cefd4","Type":"ContainerDied","Data":"763e65a77a7dd9d16e013884327002e676123bf57ede2af934085a7dcef45f1d"}
Oct 30 13:24:05.956332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.956301    4291 scope.go:122] "RemoveContainer" containerID="3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3"
Oct 30 13:24:05.956688 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:24:05.956668    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3\": container with ID starting with 3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3 not found: ID does not exist" containerID="3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3"
Oct 30 13:24:05.956739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.956697    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3"} err="failed to get container status \"3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3\": rpc error: code = NotFound desc = could not find container \"3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3\": container with ID starting with 3e98196fc0048d35d8d52bfdf1a81587784aad27ba16ef4ad54d6875be257db3 not found: ID does not exist"
Oct 30 13:24:05.956739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.956722    4291 scope.go:122] "RemoveContainer" containerID="bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356"
Oct 30 13:24:05.972182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.972153    4291 scope.go:122] "RemoveContainer" containerID="bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356"
Oct 30 13:24:05.972806 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:24:05.972780    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356\": container with ID starting with bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356 not found: ID does not exist" containerID="bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356"
Oct 30 13:24:05.972893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:05.972816    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356"} err="failed to get container status \"bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356\": rpc error: code = NotFound desc = could not find container \"bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356\": container with ID starting with bd1015be64b50575f21d133998e2b3ded9e51d724839318d20eeccf3ce798356 not found: ID does not exist"
Oct 30 13:24:06.005121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:06.005074    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"]
Oct 30 13:24:06.006903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:06.006877    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-3a82db-55cf74758c-m7jjz"]
Oct 30 13:24:06.036061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:06.035905    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"]
Oct 30 13:24:06.039932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:06.039896    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-3a82db-85f45c5db-cg88r"]
Oct 30 13:24:07.881143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:07.881104    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2915871-001e-4ec6-a2d1-a5cd3fa00e09" path="/var/lib/kubelet/pods/f2915871-001e-4ec6-a2d1-a5cd3fa00e09/volumes"
Oct 30 13:24:07.881794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:07.881769    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff9714eb-21b0-479f-9ed1-4152ce0cefd4" path="/var/lib/kubelet/pods/ff9714eb-21b0-479f-9ed1-4152ce0cefd4/volumes"
Oct 30 13:24:08.212973 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:24:08.212882    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:24:10.602824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.602797    4291 scope.go:122] "RemoveContainer" containerID="fedac7dc1fd3cd41e70b01bec32af5034eac9c78f0e95c925bad368af7c892b4"
Oct 30 13:24:10.612925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.612902    4291 scope.go:122] "RemoveContainer" containerID="dd7db99aa213167bc8b5786e228eb4bd14d81d7f217a8c2d835ff63c25fa4078"
Oct 30 13:24:10.622374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.622349    4291 scope.go:122] "RemoveContainer" containerID="21e898b8ee1cbebae95c304c61f43cf4d6c766660d545f01d088d81c6abe1495"
Oct 30 13:24:10.631768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.631745    4291 scope.go:122] "RemoveContainer" containerID="bb0bcdb0a889548d52819ae49e63bef7b4bcab257ca079b35d012278ce185362"
Oct 30 13:24:10.641032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.641007    4291 scope.go:122] "RemoveContainer" containerID="b636d05e7edef43abc2d23151ba5c418fc94ba52960eaf3eeb071f6a12d63dae"
Oct 30 13:24:10.650600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.650573    4291 scope.go:122] "RemoveContainer" containerID="c753cfafe45b3b40092bd39bc1509284dd18c2d55128213617bf31b11ae88b8c"
Oct 30 13:24:10.660173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.660152    4291 scope.go:122] "RemoveContainer" containerID="00a159bd3b28660b7048727490f71cce19f32ce553d419899e16e919ae8b77e2"
Oct 30 13:24:10.669703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.669683    4291 scope.go:122] "RemoveContainer" containerID="b815e3ccd9c401ae098e77ae1435d5e0039aefd28cdc4f7d647bfbdffffe9077"
Oct 30 13:24:10.678982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.678961    4291 scope.go:122] "RemoveContainer" containerID="5519af050eb1f78b62a5c85477b96482c57e5d3552f6acd63990d17d191ae64d"
Oct 30 13:24:10.688090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.688069    4291 scope.go:122] "RemoveContainer" containerID="2325c4727d27ff55030ddf385300cb291b22b389335d3a1758d63347933f249f"
Oct 30 13:24:10.697169 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.697143    4291 scope.go:122] "RemoveContainer" containerID="473bd3eed5b66706d777470131c0ac00db707f1f25d00eef42547bac1e84f27b"
Oct 30 13:24:10.908286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.908203    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:24:10.908888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:10.908868    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:24:17.043489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:17.043450    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"]
Oct 30 13:24:17.057307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:17.057272    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d72275a-2lpr9"]
Oct 30 13:24:17.880072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:17.880032    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6dec60b5-b084-4df0-84dd-8bc949a77d32" path="/var/lib/kubelet/pods/6dec60b5-b084-4df0-84dd-8bc949a77d32/volumes"
Oct 30 13:24:27.032675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:27.032620    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"]
Oct 30 13:24:27.044604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:27.044573    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd613a5e-8rf66"]
Oct 30 13:24:27.880507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:27.880466    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19a69509-7ff0-4be0-a1c1-8ce7e07573c2" path="/var/lib/kubelet/pods/19a69509-7ff0-4be0-a1c1-8ce7e07573c2/volumes"
Oct 30 13:24:31.671061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:31.661892    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:24:33.285757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.285716    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"]
Oct 30 13:24:33.286585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.286566    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ff9714eb-21b0-479f-9ed1-4152ce0cefd4" containerName="payload-processing"
Oct 30 13:24:33.286585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.286586    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff9714eb-21b0-479f-9ed1-4152ce0cefd4" containerName="payload-processing"
Oct 30 13:24:33.286700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.286622    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f2915871-001e-4ec6-a2d1-a5cd3fa00e09" containerName="payload-pre-processing"
Oct 30 13:24:33.286700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.286627    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2915871-001e-4ec6-a2d1-a5cd3fa00e09" containerName="payload-pre-processing"
Oct 30 13:24:33.286779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.286734    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2915871-001e-4ec6-a2d1-a5cd3fa00e09" containerName="payload-pre-processing"
Oct 30 13:24:33.286779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.286746    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff9714eb-21b0-479f-9ed1-4152ce0cefd4" containerName="payload-processing"
Oct 30 13:24:33.291302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.291283    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.307972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.307942    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"]
Oct 30 13:24:33.316716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.314824    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:33.316716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.314866    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:24:33.317140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.317103    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:24:33.317374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.317341    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:24:33.374281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.374239    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6nm8\" (UniqueName: \"kubernetes.io/projected/529ec5ac-9bd5-4381-8134-66914b8f52d6-kube-api-access-h6nm8\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.374508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.374338    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.374594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.374568    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.476023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.475981    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.476237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.476073    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6nm8\" (UniqueName: \"kubernetes.io/projected/529ec5ac-9bd5-4381-8134-66914b8f52d6-kube-api-access-h6nm8\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.476237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.476127    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.476649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.476611    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.478451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.478432    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.494297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.494273    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6nm8\" (UniqueName: \"kubernetes.io/projected/529ec5ac-9bd5-4381-8134-66914b8f52d6-kube-api-access-h6nm8\") pod \"maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.606088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.606042    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:33.915412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:33.915381    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"]
Oct 30 13:24:33.916861 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:24:33.916837    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529ec5ac_9bd5_4381_8134_66914b8f52d6.slice/crio-52ecc48c3f333396ba4eab931b5fd610839adf3c390c312dcd3c48e5d646a838 WatchSource:0}: Error finding container 52ecc48c3f333396ba4eab931b5fd610839adf3c390c312dcd3c48e5d646a838: Status 404 returned error can't find the container with id 52ecc48c3f333396ba4eab931b5fd610839adf3c390c312dcd3c48e5d646a838
Oct 30 13:24:34.191482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:34.191446    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5" event={"ID":"529ec5ac-9bd5-4381-8134-66914b8f52d6","Type":"ContainerStarted","Data":"52ecc48c3f333396ba4eab931b5fd610839adf3c390c312dcd3c48e5d646a838"}
Oct 30 13:24:35.199847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:35.199760    4291 generic.go:356] "Generic (PLEG): container finished" podID="529ec5ac-9bd5-4381-8134-66914b8f52d6" containerID="06f20999915b2456cedc76ad1ac9c511d6822fcf2f49e5902a175dc4d812a81a" exitCode=0
Oct 30 13:24:35.200262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:35.199843    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5" event={"ID":"529ec5ac-9bd5-4381-8134-66914b8f52d6","Type":"ContainerDied","Data":"06f20999915b2456cedc76ad1ac9c511d6822fcf2f49e5902a175dc4d812a81a"}
Oct 30 13:24:36.384888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.384859    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:36.518316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.518228    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529ec5ac-9bd5-4381-8134-66914b8f52d6-kube-api-access-h6nm8\" (UniqueName: \"kubernetes.io/projected/529ec5ac-9bd5-4381-8134-66914b8f52d6-kube-api-access-h6nm8\") pod \"529ec5ac-9bd5-4381-8134-66914b8f52d6\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") "
Oct 30 13:24:36.518494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.518333    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-openshift-ca\") pod \"529ec5ac-9bd5-4381-8134-66914b8f52d6\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") "
Oct 30 13:24:36.518494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.518402    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-xks-ca\") pod \"529ec5ac-9bd5-4381-8134-66914b8f52d6\" (UID: \"529ec5ac-9bd5-4381-8134-66914b8f52d6\") "
Oct 30 13:24:36.518810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.518779    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-openshift-ca" pod "529ec5ac-9bd5-4381-8134-66914b8f52d6" (UID: "529ec5ac-9bd5-4381-8134-66914b8f52d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:36.520405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.520373    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-xks-ca" pod "529ec5ac-9bd5-4381-8134-66914b8f52d6" (UID: "529ec5ac-9bd5-4381-8134-66914b8f52d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:36.520505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.520459    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529ec5ac-9bd5-4381-8134-66914b8f52d6-kube-api-access-h6nm8" pod "529ec5ac-9bd5-4381-8134-66914b8f52d6" (UID: "529ec5ac-9bd5-4381-8134-66914b8f52d6"). InnerVolumeSpecName "kube-api-access-h6nm8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:36.620169 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.620119    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:36.620169 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.620164    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/529ec5ac-9bd5-4381-8134-66914b8f52d6-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:36.620169 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:36.620175    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6nm8\" (UniqueName: \"kubernetes.io/projected/529ec5ac-9bd5-4381-8134-66914b8f52d6-kube-api-access-h6nm8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:37.215563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:37.215530    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"
Oct 30 13:24:37.215563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:37.215548    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5" event={"ID":"529ec5ac-9bd5-4381-8134-66914b8f52d6","Type":"ContainerDied","Data":"52ecc48c3f333396ba4eab931b5fd610839adf3c390c312dcd3c48e5d646a838"}
Oct 30 13:24:37.215817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:37.215601    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52ecc48c3f333396ba4eab931b5fd610839adf3c390c312dcd3c48e5d646a838"
Oct 30 13:24:38.212728 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:24:38.212691    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:24:48.483536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:48.483492    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"]
Oct 30 13:24:48.493088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:48.493061    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-121d75-t2k5d"]
Oct 30 13:24:49.879312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:49.879278    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c12b5bbd-8843-40bc-b0b9-8635951f49bc" path="/var/lib/kubelet/pods/c12b5bbd-8843-40bc-b0b9-8635951f49bc/volumes"
Oct 30 13:24:50.305495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.305398    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"]
Oct 30 13:24:50.305711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.305674    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf" podUID="48935990-b8d4-4d88-a661-0eb0a581e2e1" containerName="payload-processing" containerID="cri-o://455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee" gracePeriod=30
Oct 30 13:24:50.371323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.371283    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"]
Oct 30 13:24:50.371612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.371556    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k" podUID="ff8dc794-a677-4869-b002-6a1679093f57" containerName="payload-pre-processing" containerID="cri-o://36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf" gracePeriod=30
Oct 30 13:24:50.863948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.863916    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:24:50.867066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.867040    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:24:50.961092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.961017    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48935990-b8d4-4d88-a661-0eb0a581e2e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48935990-b8d4-4d88-a661-0eb0a581e2e1-plugins-config-volume\") pod \"48935990-b8d4-4d88-a661-0eb0a581e2e1\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") "
Oct 30 13:24:50.961567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.961100    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48935990-b8d4-4d88-a661-0eb0a581e2e1-kube-api-access-gst48\" (UniqueName: \"kubernetes.io/projected/48935990-b8d4-4d88-a661-0eb0a581e2e1-kube-api-access-gst48\") pod \"48935990-b8d4-4d88-a661-0eb0a581e2e1\" (UID: \"48935990-b8d4-4d88-a661-0eb0a581e2e1\") "
Oct 30 13:24:50.961567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.961152    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff8dc794-a677-4869-b002-6a1679093f57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8dc794-a677-4869-b002-6a1679093f57-plugins-config-volume\") pod \"ff8dc794-a677-4869-b002-6a1679093f57\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") "
Oct 30 13:24:50.961567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.961204    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff8dc794-a677-4869-b002-6a1679093f57-kube-api-access-6d56w\" (UniqueName: \"kubernetes.io/projected/ff8dc794-a677-4869-b002-6a1679093f57-kube-api-access-6d56w\") pod \"ff8dc794-a677-4869-b002-6a1679093f57\" (UID: \"ff8dc794-a677-4869-b002-6a1679093f57\") "
Oct 30 13:24:50.961567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.961538    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48935990-b8d4-4d88-a661-0eb0a581e2e1-plugins-config-volume" pod "48935990-b8d4-4d88-a661-0eb0a581e2e1" (UID: "48935990-b8d4-4d88-a661-0eb0a581e2e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:50.961567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.961565    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff8dc794-a677-4869-b002-6a1679093f57-plugins-config-volume" pod "ff8dc794-a677-4869-b002-6a1679093f57" (UID: "ff8dc794-a677-4869-b002-6a1679093f57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:50.962158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.962094    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48935990-b8d4-4d88-a661-0eb0a581e2e1-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:50.962158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.962135    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff8dc794-a677-4869-b002-6a1679093f57-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:50.963466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.963438    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48935990-b8d4-4d88-a661-0eb0a581e2e1-kube-api-access-gst48" pod "48935990-b8d4-4d88-a661-0eb0a581e2e1" (UID: "48935990-b8d4-4d88-a661-0eb0a581e2e1"). InnerVolumeSpecName "kube-api-access-gst48". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:50.963801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:50.963778    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff8dc794-a677-4869-b002-6a1679093f57-kube-api-access-6d56w" pod "ff8dc794-a677-4869-b002-6a1679093f57" (UID: "ff8dc794-a677-4869-b002-6a1679093f57"). InnerVolumeSpecName "kube-api-access-6d56w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:51.061099 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.061062    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"]
Oct 30 13:24:51.063455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.063427    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gst48\" (UniqueName: \"kubernetes.io/projected/48935990-b8d4-4d88-a661-0eb0a581e2e1-kube-api-access-gst48\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:51.063455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.063452    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d56w\" (UniqueName: \"kubernetes.io/projected/ff8dc794-a677-4869-b002-6a1679093f57-kube-api-access-6d56w\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:51.064275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.064249    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-5dcf84-44sx5"]
Oct 30 13:24:51.316797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.316706    4291 generic.go:356] "Generic (PLEG): container finished" podID="48935990-b8d4-4d88-a661-0eb0a581e2e1" containerID="455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee" exitCode=0
Oct 30 13:24:51.316797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.316772    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"
Oct 30 13:24:51.317018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.316797    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf" event={"ID":"48935990-b8d4-4d88-a661-0eb0a581e2e1","Type":"ContainerDied","Data":"455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee"}
Oct 30 13:24:51.317018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.316858    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf" event={"ID":"48935990-b8d4-4d88-a661-0eb0a581e2e1","Type":"ContainerDied","Data":"55b37d04706e3804995e2bed99946fd2275382d2d3a5d1564a010d77a3de1e3c"}
Oct 30 13:24:51.317018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.316889    4291 scope.go:122] "RemoveContainer" containerID="455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee"
Oct 30 13:24:51.318302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.318204    4291 generic.go:356] "Generic (PLEG): container finished" podID="ff8dc794-a677-4869-b002-6a1679093f57" containerID="36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf" exitCode=0
Oct 30 13:24:51.318302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.318261    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"
Oct 30 13:24:51.318408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.318334    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k" event={"ID":"ff8dc794-a677-4869-b002-6a1679093f57","Type":"ContainerDied","Data":"36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf"}
Oct 30 13:24:51.318408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.318368    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k" event={"ID":"ff8dc794-a677-4869-b002-6a1679093f57","Type":"ContainerDied","Data":"34268aa7baf4e351e50c44c38b35230ec5c4558fa84c42cbeab2b34dcb0163cf"}
Oct 30 13:24:51.342009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.341993    4291 scope.go:122] "RemoveContainer" containerID="455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee"
Oct 30 13:24:51.342313 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:24:51.342296    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee\": container with ID starting with 455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee not found: ID does not exist" containerID="455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee"
Oct 30 13:24:51.342387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.342319    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee"} err="failed to get container status \"455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee\": rpc error: code = NotFound desc = could not find container \"455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee\": container with ID starting with 455014496a1ed3c83e231bdb46387f9322139175c7c5a8c78b501b7e817a7bee not found: ID does not exist"
Oct 30 13:24:51.342387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.342335    4291 scope.go:122] "RemoveContainer" containerID="36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf"
Oct 30 13:24:51.352494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.352474    4291 scope.go:122] "RemoveContainer" containerID="36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf"
Oct 30 13:24:51.352801 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:24:51.352758    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf\": container with ID starting with 36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf not found: ID does not exist" containerID="36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf"
Oct 30 13:24:51.352888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.352806    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf"} err="failed to get container status \"36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf\": rpc error: code = NotFound desc = could not find container \"36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf\": container with ID starting with 36c86f30c67762fb6b813e10062178534c80c4c7bde087ae03f7ff0bc08736cf not found: ID does not exist"
Oct 30 13:24:51.381413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.381388    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"]
Oct 30 13:24:51.384486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.384467    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-5dcf84-5898d74695-nnz2k"]
Oct 30 13:24:51.414980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.414949    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"]
Oct 30 13:24:51.418403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.418381    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-5dcf84-7d45b46f78-bh5nf"]
Oct 30 13:24:51.879397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.879364    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48935990-b8d4-4d88-a661-0eb0a581e2e1" path="/var/lib/kubelet/pods/48935990-b8d4-4d88-a661-0eb0a581e2e1/volumes"
Oct 30 13:24:51.879865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.879849    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529ec5ac-9bd5-4381-8134-66914b8f52d6" path="/var/lib/kubelet/pods/529ec5ac-9bd5-4381-8134-66914b8f52d6/volumes"
Oct 30 13:24:51.880220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:51.880207    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff8dc794-a677-4869-b002-6a1679093f57" path="/var/lib/kubelet/pods/ff8dc794-a677-4869-b002-6a1679093f57/volumes"
Oct 30 13:24:53.274821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.274602    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"]
Oct 30 13:24:53.276106 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276081    4291 cpu_manager.go:414] "containerMap: removing container" podUID="529ec5ac-9bd5-4381-8134-66914b8f52d6" containerName="revoke-keys"
Oct 30 13:24:53.276249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276236    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529ec5ac-9bd5-4381-8134-66914b8f52d6" containerName="revoke-keys"
Oct 30 13:24:53.276358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276348    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ff8dc794-a677-4869-b002-6a1679093f57" containerName="payload-pre-processing"
Oct 30 13:24:53.276426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276417    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff8dc794-a677-4869-b002-6a1679093f57" containerName="payload-pre-processing"
Oct 30 13:24:53.276511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276503    4291 cpu_manager.go:414] "containerMap: removing container" podUID="48935990-b8d4-4d88-a661-0eb0a581e2e1" containerName="payload-processing"
Oct 30 13:24:53.276581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276572    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48935990-b8d4-4d88-a661-0eb0a581e2e1" containerName="payload-processing"
Oct 30 13:24:53.276843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276830    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="48935990-b8d4-4d88-a661-0eb0a581e2e1" containerName="payload-processing"
Oct 30 13:24:53.276947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.276937    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="529ec5ac-9bd5-4381-8134-66914b8f52d6" containerName="revoke-keys"
Oct 30 13:24:53.277027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.277018    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff8dc794-a677-4869-b002-6a1679093f57" containerName="payload-pre-processing"
Oct 30 13:24:53.282729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.282708    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.297063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.297034    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"]
Oct 30 13:24:53.304325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.304298    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:24:53.304498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.304309    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:24:53.304498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.304353    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:24:53.309256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.309238    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:24:53.403121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.403089    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.403121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.403124    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.403324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.403208    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pxg7\" (UniqueName: \"kubernetes.io/projected/d6325be5-9c9c-499a-a8c3-ab4850624fd9-kube-api-access-7pxg7\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.504802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.504765    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.504802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.504802    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.505014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.504851    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pxg7\" (UniqueName: \"kubernetes.io/projected/d6325be5-9c9c-499a-a8c3-ab4850624fd9-kube-api-access-7pxg7\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.505510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.505490    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.507217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.507196    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.523380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.523352    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pxg7\" (UniqueName: \"kubernetes.io/projected/d6325be5-9c9c-499a-a8c3-ab4850624fd9-kube-api-access-7pxg7\") pod \"maas-api-revoke-keys-e2e-api-a-c9e946-m55b7\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.601878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.601844    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:53.910163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:53.910133    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"]
Oct 30 13:24:53.911629 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:24:53.911607    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6325be5_9c9c_499a_a8c3_ab4850624fd9.slice/crio-4a1d572465485c0b947989a5a49ad603dd1a4727a951afa213757bc4c51478ad WatchSource:0}: Error finding container 4a1d572465485c0b947989a5a49ad603dd1a4727a951afa213757bc4c51478ad: Status 404 returned error can't find the container with id 4a1d572465485c0b947989a5a49ad603dd1a4727a951afa213757bc4c51478ad
Oct 30 13:24:54.342735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:54.342698    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7" event={"ID":"d6325be5-9c9c-499a-a8c3-ab4850624fd9","Type":"ContainerStarted","Data":"312859af663593eb18c07dad75769ef787c77169f0d4522477204d4798b2326b"}
Oct 30 13:24:54.343135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:54.342763    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7" event={"ID":"d6325be5-9c9c-499a-a8c3-ab4850624fd9","Type":"ContainerStarted","Data":"4a1d572465485c0b947989a5a49ad603dd1a4727a951afa213757bc4c51478ad"}
Oct 30 13:24:54.383128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:54.383069    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7" podStartSLOduration=1.383052754 podStartE2EDuration="1.383052754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:24:54.379612686 +0000 UTC m=+1304.639850920" watchObservedRunningTime="2026-10-30 13:24:54.383052754 +0000 UTC m=+1304.643290987"
Oct 30 13:24:55.351769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:55.351728    4291 generic.go:356] "Generic (PLEG): container finished" podID="d6325be5-9c9c-499a-a8c3-ab4850624fd9" containerID="312859af663593eb18c07dad75769ef787c77169f0d4522477204d4798b2326b" exitCode=0
Oct 30 13:24:55.352141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:55.351812    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7" event={"ID":"d6325be5-9c9c-499a-a8c3-ab4850624fd9","Type":"ContainerDied","Data":"312859af663593eb18c07dad75769ef787c77169f0d4522477204d4798b2326b"}
Oct 30 13:24:56.112451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.107908    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"]
Oct 30 13:24:56.112451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.108157    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy" containerID="cri-o://fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410" gracePeriod=30
Oct 30 13:24:56.599776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.599748    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:56.708584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.708490    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"]
Oct 30 13:24:56.709162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.709145    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d6325be5-9c9c-499a-a8c3-ab4850624fd9" containerName="revoke-keys"
Oct 30 13:24:56.709162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.709162    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6325be5-9c9c-499a-a8c3-ab4850624fd9" containerName="revoke-keys"
Oct 30 13:24:56.709314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.709267    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6325be5-9c9c-499a-a8c3-ab4850624fd9" containerName="revoke-keys"
Oct 30 13:24:56.712678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.712650    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.729535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.729497    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"]
Oct 30 13:24:56.760854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.760816    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6325be5-9c9c-499a-a8c3-ab4850624fd9-kube-api-access-7pxg7\" (UniqueName: \"kubernetes.io/projected/d6325be5-9c9c-499a-a8c3-ab4850624fd9-kube-api-access-7pxg7\") pod \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") "
Oct 30 13:24:56.761053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.760866    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-openshift-ca\") pod \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") "
Oct 30 13:24:56.761053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.760943    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-xks-ca\") pod \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\" (UID: \"d6325be5-9c9c-499a-a8c3-ab4850624fd9\") "
Oct 30 13:24:56.761678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.761603    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-openshift-ca" pod "d6325be5-9c9c-499a-a8c3-ab4850624fd9" (UID: "d6325be5-9c9c-499a-a8c3-ab4850624fd9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:56.767621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.767534    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-xks-ca" pod "d6325be5-9c9c-499a-a8c3-ab4850624fd9" (UID: "d6325be5-9c9c-499a-a8c3-ab4850624fd9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:56.767946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.767917    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6325be5-9c9c-499a-a8c3-ab4850624fd9-kube-api-access-7pxg7" pod "d6325be5-9c9c-499a-a8c3-ab4850624fd9" (UID: "d6325be5-9c9c-499a-a8c3-ab4850624fd9"). InnerVolumeSpecName "kube-api-access-7pxg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:56.862962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.862910    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.863144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.863067    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppqdd\" (UniqueName: \"kubernetes.io/projected/89b475c1-33e5-46e3-98c4-69bdaaabde56-kube-api-access-ppqdd\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.863191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.863144    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.863254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.863241    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:56.863289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.863258    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pxg7\" (UniqueName: \"kubernetes.io/projected/d6325be5-9c9c-499a-a8c3-ab4850624fd9-kube-api-access-7pxg7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:56.863289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.863267    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6325be5-9c9c-499a-a8c3-ab4850624fd9-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:56.964533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.964445    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppqdd\" (UniqueName: \"kubernetes.io/projected/89b475c1-33e5-46e3-98c4-69bdaaabde56-kube-api-access-ppqdd\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.964735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.964532    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.964735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.964723    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.965349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.965324    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.966922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.966894    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:56.983131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:56.983103    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppqdd\" (UniqueName: \"kubernetes.io/projected/89b475c1-33e5-46e3-98c4-69bdaaabde56-kube-api-access-ppqdd\") pod \"maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:57.030939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:57.030899    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:57.343137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:57.343104    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"]
Oct 30 13:24:57.344338 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:24:57.344313    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b475c1_33e5_46e3_98c4_69bdaaabde56.slice/crio-2dd8d05a90bad0d893e044d7f00685c4caa1f3e27771214cb9373c17dfab91c9 WatchSource:0}: Error finding container 2dd8d05a90bad0d893e044d7f00685c4caa1f3e27771214cb9373c17dfab91c9: Status 404 returned error can't find the container with id 2dd8d05a90bad0d893e044d7f00685c4caa1f3e27771214cb9373c17dfab91c9
Oct 30 13:24:57.378476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:57.378445    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"
Oct 30 13:24:57.378693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:57.378478    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7" event={"ID":"d6325be5-9c9c-499a-a8c3-ab4850624fd9","Type":"ContainerDied","Data":"4a1d572465485c0b947989a5a49ad603dd1a4727a951afa213757bc4c51478ad"}
Oct 30 13:24:57.378693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:57.378531    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a1d572465485c0b947989a5a49ad603dd1a4727a951afa213757bc4c51478ad"
Oct 30 13:24:57.380357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:57.380326    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw" event={"ID":"89b475c1-33e5-46e3-98c4-69bdaaabde56","Type":"ContainerStarted","Data":"2dd8d05a90bad0d893e044d7f00685c4caa1f3e27771214cb9373c17dfab91c9"}
Oct 30 13:24:58.011652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:58.011542    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:24:58.389409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:58.389371    4291 generic.go:356] "Generic (PLEG): container finished" podID="89b475c1-33e5-46e3-98c4-69bdaaabde56" containerID="9fb4cdd60f8e1458573e23385332de051b5a0ab7f00845f77e1f4950def9f28b" exitCode=0
Oct 30 13:24:58.389570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:58.389418    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw" event={"ID":"89b475c1-33e5-46e3-98c4-69bdaaabde56","Type":"ContainerDied","Data":"9fb4cdd60f8e1458573e23385332de051b5a0ab7f00845f77e1f4950def9f28b"}
Oct 30 13:24:59.577659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.574422    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"]
Oct 30 13:24:59.595861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.595832    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"]
Oct 30 13:24:59.596204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.596184    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.722620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.722564    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.722812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.722674    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7tnx\" (UniqueName: \"kubernetes.io/projected/682ccddd-dcf2-4cba-a805-669cb2cfcf52-kube-api-access-l7tnx\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.722812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.722741    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.734164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.734127    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:24:59.827713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.827676    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-openshift-ca\") pod \"89b475c1-33e5-46e3-98c4-69bdaaabde56\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") "
Oct 30 13:24:59.827930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.827782    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89b475c1-33e5-46e3-98c4-69bdaaabde56-kube-api-access-ppqdd\" (UniqueName: \"kubernetes.io/projected/89b475c1-33e5-46e3-98c4-69bdaaabde56-kube-api-access-ppqdd\") pod \"89b475c1-33e5-46e3-98c4-69bdaaabde56\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") "
Oct 30 13:24:59.828000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.827955    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-xks-ca\") pod \"89b475c1-33e5-46e3-98c4-69bdaaabde56\" (UID: \"89b475c1-33e5-46e3-98c4-69bdaaabde56\") "
Oct 30 13:24:59.828210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.828173    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-openshift-ca" pod "89b475c1-33e5-46e3-98c4-69bdaaabde56" (UID: "89b475c1-33e5-46e3-98c4-69bdaaabde56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:24:59.828418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.828394    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.828553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.828436    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7tnx\" (UniqueName: \"kubernetes.io/projected/682ccddd-dcf2-4cba-a805-669cb2cfcf52-kube-api-access-l7tnx\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.828553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.828491    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.828683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.828592    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:59.829354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.829322    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.830651 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.830610    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-xks-ca" pod "89b475c1-33e5-46e3-98c4-69bdaaabde56" (UID: "89b475c1-33e5-46e3-98c4-69bdaaabde56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:24:59.830809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.830658    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89b475c1-33e5-46e3-98c4-69bdaaabde56-kube-api-access-ppqdd" pod "89b475c1-33e5-46e3-98c4-69bdaaabde56" (UID: "89b475c1-33e5-46e3-98c4-69bdaaabde56"). InnerVolumeSpecName "kube-api-access-ppqdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:24:59.831726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.831705    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.848833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.848795    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7tnx\" (UniqueName: \"kubernetes.io/projected/682ccddd-dcf2-4cba-a805-669cb2cfcf52-kube-api-access-l7tnx\") pod \"maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:24:59.930440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.930342    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89b475c1-33e5-46e3-98c4-69bdaaabde56-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:24:59.930440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:24:59.930377    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppqdd\" (UniqueName: \"kubernetes.io/projected/89b475c1-33e5-46e3-98c4-69bdaaabde56-kube-api-access-ppqdd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:00.030513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:00.030470    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:25:00.383744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:00.383705    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"]
Oct 30 13:25:00.438453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:00.438424    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"
Oct 30 13:25:00.438453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:00.438444    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw" event={"ID":"89b475c1-33e5-46e3-98c4-69bdaaabde56","Type":"ContainerDied","Data":"2dd8d05a90bad0d893e044d7f00685c4caa1f3e27771214cb9373c17dfab91c9"}
Oct 30 13:25:00.438618 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:00.438500    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2dd8d05a90bad0d893e044d7f00685c4caa1f3e27771214cb9373c17dfab91c9"
Oct 30 13:25:00.440203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:00.440179    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d" event={"ID":"682ccddd-dcf2-4cba-a805-669cb2cfcf52","Type":"ContainerStarted","Data":"a3887238a93726fa7e1dbc3ff50461ffc53877aaa8d0ac0f9ff2d1634caff4a1"}
Oct 30 13:25:01.442321 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.442293    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:25:01.449466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.449430    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d" event={"ID":"682ccddd-dcf2-4cba-a805-669cb2cfcf52","Type":"ContainerStarted","Data":"a402e5c6a07e026dcc7c9b02153e032b817f05534dbb53a66b9f62ccc2383a4a"}
Oct 30 13:25:01.481382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.480212    4291 generic.go:356] "Generic (PLEG): container finished" podID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerID="fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410" exitCode=0
Oct 30 13:25:01.481382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.480350    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" event={"ID":"533c0e6f-0446-40d9-b4fa-58fbd1c34c59","Type":"ContainerDied","Data":"fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410"}
Oct 30 13:25:01.481382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.480404    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl" event={"ID":"533c0e6f-0446-40d9-b4fa-58fbd1c34c59","Type":"ContainerDied","Data":"ae41379bbd217069630602c4eb40873aa81ce059d6a87c663cdf75f0d4bdd572"}
Oct 30 13:25:01.481382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.480428    4291 scope.go:122] "RemoveContainer" containerID="fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410"
Oct 30 13:25:01.481382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.480591    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"
Oct 30 13:25:01.493777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.493752    4291 scope.go:122] "RemoveContainer" containerID="ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34"
Oct 30 13:25:01.512549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.512516    4291 scope.go:122] "RemoveContainer" containerID="fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410"
Oct 30 13:25:01.513072 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:25:01.513043    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410\": container with ID starting with fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410 not found: ID does not exist" containerID="fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410"
Oct 30 13:25:01.513161 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.513084    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410"} err="failed to get container status \"fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410\": rpc error: code = NotFound desc = could not find container \"fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410\": container with ID starting with fa2f1816258224c2aeb66bcf0647f078fe255064967f29402d1c1d125a6a2410 not found: ID does not exist"
Oct 30 13:25:01.513161 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.513111    4291 scope.go:122] "RemoveContainer" containerID="ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34"
Oct 30 13:25:01.519718 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:25:01.519620    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34\": container with ID starting with ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34 not found: ID does not exist" containerID="ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34"
Oct 30 13:25:01.519871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.519723    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34"} err="failed to get container status \"ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34\": rpc error: code = NotFound desc = could not find container \"ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34\": container with ID starting with ee2d7ad7ce28d996dad2aab0dfc63f2d0c0069d8b217dd1c57e1f1ab59d37a34 not found: ID does not exist"
Oct 30 13:25:01.578866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.577749    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d" podStartSLOduration=2.5777244489999998 podStartE2EDuration="2.577724449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:25:01.547890871 +0000 UTC m=+1311.808129108" watchObservedRunningTime="2026-10-30 13:25:01.577724449 +0000 UTC m=+1311.837962686"
Oct 30 13:25:01.582309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582279    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-data\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.582470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582339    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-socket\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.582470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582432    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istiod-ca-cert\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.582470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582461    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-envoy\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.582660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582554    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-credential-socket\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.582660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582589    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-socket" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:01.582825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582799    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-credential-socket" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:01.582886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582851    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-podinfo\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.582937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582884    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-token\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-token\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.582937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582907    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istiod-ca-cert" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:01.583041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.582966    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-kube-api-access-xth85\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-kube-api-access-xth85\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.583193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.583168    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-certs\") pod \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\" (UID: \"533c0e6f-0446-40d9-b4fa-58fbd1c34c59\") "
Oct 30 13:25:01.583282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.583183    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-data" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:01.583794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.583772    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.583896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.583799    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.583896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.583811    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.583896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.583821    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.584209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.584176    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-certs" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:01.585386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.585354    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-envoy" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:25:01.585457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.585388    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-podinfo" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:25:01.586117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.586091    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-token" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:01.586206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.586113    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-kube-api-access-xth85" pod "533c0e6f-0446-40d9-b4fa-58fbd1c34c59" (UID: "533c0e6f-0446-40d9-b4fa-58fbd1c34c59"). InnerVolumeSpecName "kube-api-access-xth85". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:01.685982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.685942    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.685982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.685983    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.686230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.685997    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xth85\" (UniqueName: \"kubernetes.io/projected/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-kube-api-access-xth85\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.686230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.686014    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.686230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.686028    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/533c0e6f-0446-40d9-b4fa-58fbd1c34c59-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:01.838906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.838808    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"]
Oct 30 13:25:01.843186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.843157    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-121d75-openshift-default-75d9d7748f-bwwzl"]
Oct 30 13:25:01.886736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:01.886703    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" path="/var/lib/kubelet/pods/533c0e6f-0446-40d9-b4fa-58fbd1c34c59/volumes"
Oct 30 13:25:02.491276 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:02.491238    4291 generic.go:356] "Generic (PLEG): container finished" podID="682ccddd-dcf2-4cba-a805-669cb2cfcf52" containerID="a402e5c6a07e026dcc7c9b02153e032b817f05534dbb53a66b9f62ccc2383a4a" exitCode=0
Oct 30 13:25:02.491822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:02.491318    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d" event={"ID":"682ccddd-dcf2-4cba-a805-669cb2cfcf52","Type":"ContainerDied","Data":"a402e5c6a07e026dcc7c9b02153e032b817f05534dbb53a66b9f62ccc2383a4a"}
Oct 30 13:25:03.934075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:03.934043    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:25:04.034159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.034122    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-xks-ca\") pod \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") "
Oct 30 13:25:04.034364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.034340    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-openshift-ca\") pod \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") "
Oct 30 13:25:04.034473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.034423    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/682ccddd-dcf2-4cba-a805-669cb2cfcf52-kube-api-access-l7tnx\" (UniqueName: \"kubernetes.io/projected/682ccddd-dcf2-4cba-a805-669cb2cfcf52-kube-api-access-l7tnx\") pod \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\" (UID: \"682ccddd-dcf2-4cba-a805-669cb2cfcf52\") "
Oct 30 13:25:04.034781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.034755    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-openshift-ca" pod "682ccddd-dcf2-4cba-a805-669cb2cfcf52" (UID: "682ccddd-dcf2-4cba-a805-669cb2cfcf52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:04.035117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.035094    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:04.036422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.036399    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-xks-ca" pod "682ccddd-dcf2-4cba-a805-669cb2cfcf52" (UID: "682ccddd-dcf2-4cba-a805-669cb2cfcf52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:04.036584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.036565    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/682ccddd-dcf2-4cba-a805-669cb2cfcf52-kube-api-access-l7tnx" pod "682ccddd-dcf2-4cba-a805-669cb2cfcf52" (UID: "682ccddd-dcf2-4cba-a805-669cb2cfcf52"). InnerVolumeSpecName "kube-api-access-l7tnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:04.139078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.138949    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7tnx\" (UniqueName: \"kubernetes.io/projected/682ccddd-dcf2-4cba-a805-669cb2cfcf52-kube-api-access-l7tnx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:04.139303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.139281    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/682ccddd-dcf2-4cba-a805-669cb2cfcf52-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:04.510250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.510156    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"
Oct 30 13:25:04.510250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.510188    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d" event={"ID":"682ccddd-dcf2-4cba-a805-669cb2cfcf52","Type":"ContainerDied","Data":"a3887238a93726fa7e1dbc3ff50461ffc53877aaa8d0ac0f9ff2d1634caff4a1"}
Oct 30 13:25:04.510250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:04.510240    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3887238a93726fa7e1dbc3ff50461ffc53877aaa8d0ac0f9ff2d1634caff4a1"
Oct 30 13:25:06.396043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:06.396011    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l_21a22c44-e47f-43a3-ac45-90ff7853e5ba/payload-processing/0.log"
Oct 30 13:25:07.098130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.098082    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"]
Oct 30 13:25:07.132818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.132778    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"]
Oct 30 13:25:07.987724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.987687    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"]
Oct 30 13:25:07.988246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988232    4291 cpu_manager.go:414] "containerMap: removing container" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy"
Oct 30 13:25:07.988306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988247    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy"
Oct 30 13:25:07.988306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988256    4291 cpu_manager.go:414] "containerMap: removing container" podUID="682ccddd-dcf2-4cba-a805-669cb2cfcf52" containerName="revoke-keys"
Oct 30 13:25:07.988306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988261    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="682ccddd-dcf2-4cba-a805-669cb2cfcf52" containerName="revoke-keys"
Oct 30 13:25:07.988306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988273    4291 cpu_manager.go:414] "containerMap: removing container" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy"
Oct 30 13:25:07.988306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988279    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy"
Oct 30 13:25:07.988454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988310    4291 cpu_manager.go:414] "containerMap: removing container" podUID="89b475c1-33e5-46e3-98c4-69bdaaabde56" containerName="revoke-keys"
Oct 30 13:25:07.988454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988314    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89b475c1-33e5-46e3-98c4-69bdaaabde56" containerName="revoke-keys"
Oct 30 13:25:07.988454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988412    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy"
Oct 30 13:25:07.988454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988419    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="682ccddd-dcf2-4cba-a805-669cb2cfcf52" containerName="revoke-keys"
Oct 30 13:25:07.988454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988426    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="533c0e6f-0446-40d9-b4fa-58fbd1c34c59" containerName="istio-proxy"
Oct 30 13:25:07.988454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.988436    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="89b475c1-33e5-46e3-98c4-69bdaaabde56" containerName="revoke-keys"
Oct 30 13:25:07.992116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:07.992094    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.009780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.009744    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"]
Oct 30 13:25:08.017087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.017054    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:08.017942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.017056    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:25:08.017942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.017141    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:25:08.018339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.018315    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:25:08.127416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.127379    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.127599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.127548    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.127746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.127718    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9k47\" (UniqueName: \"kubernetes.io/projected/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-kube-api-access-h9k47\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.215946 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:25:08.215894    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:25:08.229561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.229524    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.229787 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.229616    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.229787 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.229721    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9k47\" (UniqueName: \"kubernetes.io/projected/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-kube-api-access-h9k47\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.230394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.230369    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.232495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.232466    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.248884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.248792    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9k47\" (UniqueName: \"kubernetes.io/projected/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-kube-api-access-h9k47\") pod \"maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.309146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.309107    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:08.631528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:08.631496    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"]
Oct 30 13:25:08.634027 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:25:08.633995    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81a3aed1_f1e2_4d11_91c6_61a4d930d40b.slice/crio-928ba778ed091c831981d40d214223023d2cc8259241c3c3b0ea441550d311a6 WatchSource:0}: Error finding container 928ba778ed091c831981d40d214223023d2cc8259241c3c3b0ea441550d311a6: Status 404 returned error can't find the container with id 928ba778ed091c831981d40d214223023d2cc8259241c3c3b0ea441550d311a6
Oct 30 13:25:09.557694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.557576    4291 generic.go:356] "Generic (PLEG): container finished" podID="81a3aed1-f1e2-4d11-91c6-61a4d930d40b" containerID="7263aeb703d72cce83bc693af58ec21000d7349519361eaa4df8d7f7272d5a92" exitCode=0
Oct 30 13:25:09.557694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.557672    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c" event={"ID":"81a3aed1-f1e2-4d11-91c6-61a4d930d40b","Type":"ContainerDied","Data":"7263aeb703d72cce83bc693af58ec21000d7349519361eaa4df8d7f7272d5a92"}
Oct 30 13:25:09.558252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.557745    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c" event={"ID":"81a3aed1-f1e2-4d11-91c6-61a4d930d40b","Type":"ContainerStarted","Data":"928ba778ed091c831981d40d214223023d2cc8259241c3c3b0ea441550d311a6"}
Oct 30 13:25:09.559859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.559833    4291 generic.go:356] "Generic (PLEG): container finished" podID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerID="23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab" exitCode=0
Oct 30 13:25:09.559996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.559906    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" event={"ID":"50f2b6e2-8cf4-437f-97f8-573f66633fec","Type":"ContainerDied","Data":"23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab"}
Oct 30 13:25:09.560349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.560333    4291 scope.go:122] "RemoveContainer" containerID="23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab"
Oct 30 13:25:09.562008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.561977    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:25:09.562112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.562064    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:25:09.562179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:09.562114    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:25:10.033264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.033224    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"]
Oct 30 13:25:10.036442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.036413    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a94e4f-bjb7m"]
Oct 30 13:25:10.583620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.583581    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" event={"ID":"50f2b6e2-8cf4-437f-97f8-573f66633fec","Type":"ContainerStarted","Data":"c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669"}
Oct 30 13:25:10.707528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.706941    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:25:10.707702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.707547    4291 patch_prober.go:29] interesting pod/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:25:10.707702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.707594    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:25:10.788011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.787981    4291 scope.go:122] "RemoveContainer" containerID="dc415f5f3a7d398aa07f4418d5af1804351979007f2465873373a08f92c8701f"
Oct 30 13:25:10.791143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.791126    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:10.797523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.797502    4291 scope.go:122] "RemoveContainer" containerID="7bfb904866b1db1a750b079474d8053e750c66f33daafe8abb31e4a1d76908b2"
Oct 30 13:25:10.806792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.806768    4291 scope.go:122] "RemoveContainer" containerID="a71b0fab420657bcc40d0dbe7d0f7d6f35f1018e46d843621d85ed252221fd92"
Oct 30 13:25:10.816037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.816015    4291 scope.go:122] "RemoveContainer" containerID="e904d02fe2889a5a8747e1619f2b76767f2d29ac267e08b01d0d0155be8d9f02"
Oct 30 13:25:10.888810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.888031    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-kube-api-access-h9k47\" (UniqueName: \"kubernetes.io/projected/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-kube-api-access-h9k47\") pod \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") "
Oct 30 13:25:10.888810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.888114    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-xks-ca\") pod \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") "
Oct 30 13:25:10.888810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.888291    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-openshift-ca\") pod \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\" (UID: \"81a3aed1-f1e2-4d11-91c6-61a4d930d40b\") "
Oct 30 13:25:10.889138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.889103    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-openshift-ca" pod "81a3aed1-f1e2-4d11-91c6-61a4d930d40b" (UID: "81a3aed1-f1e2-4d11-91c6-61a4d930d40b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:10.891571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.891533    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-kube-api-access-h9k47" pod "81a3aed1-f1e2-4d11-91c6-61a4d930d40b" (UID: "81a3aed1-f1e2-4d11-91c6-61a4d930d40b"). InnerVolumeSpecName "kube-api-access-h9k47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:10.896964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.896032    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-xks-ca" pod "81a3aed1-f1e2-4d11-91c6-61a4d930d40b" (UID: "81a3aed1-f1e2-4d11-91c6-61a4d930d40b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:10.990204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.990170    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:10.990204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.990201    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9k47\" (UniqueName: \"kubernetes.io/projected/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-kube-api-access-h9k47\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:10.990204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:10.990212    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81a3aed1-f1e2-4d11-91c6-61a4d930d40b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:11.592145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:11.592117    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"
Oct 30 13:25:11.592545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:11.592153    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c" event={"ID":"81a3aed1-f1e2-4d11-91c6-61a4d930d40b","Type":"ContainerDied","Data":"928ba778ed091c831981d40d214223023d2cc8259241c3c3b0ea441550d311a6"}
Oct 30 13:25:11.592545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:11.592200    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="928ba778ed091c831981d40d214223023d2cc8259241c3c3b0ea441550d311a6"
Oct 30 13:25:11.707510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:11.707475    4291 patch_prober.go:29] interesting pod/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:25:11.707720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:11.707536    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:25:11.883545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:11.883515    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb61b6b8-22be-4332-bd9d-3bd93997aaf5" path="/var/lib/kubelet/pods/bb61b6b8-22be-4332-bd9d-3bd93997aaf5/volumes"
Oct 30 13:25:12.099087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.099012    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk" podUID="7a766dbd-6979-47b9-b75c-2385a43066c9" containerName="payload-processing" containerID="cri-o://ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc" gracePeriod=25
Oct 30 13:25:12.133335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.133294    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr" podUID="ca49c6f6-faa0-4a22-ad62-a552ec4484d5" containerName="payload-pre-processing" containerID="cri-o://4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241" gracePeriod=25
Oct 30 13:25:12.479694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.470841    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:25:12.507550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.507523    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:25:12.603273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.603238    4291 generic.go:356] "Generic (PLEG): container finished" podID="ca49c6f6-faa0-4a22-ad62-a552ec4484d5" containerID="4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241" exitCode=0
Oct 30 13:25:12.603809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.603397    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr" event={"ID":"ca49c6f6-faa0-4a22-ad62-a552ec4484d5","Type":"ContainerDied","Data":"4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241"}
Oct 30 13:25:12.603809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.603438    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr" event={"ID":"ca49c6f6-faa0-4a22-ad62-a552ec4484d5","Type":"ContainerDied","Data":"5aad12bf4dcd76364efa621d11af51c70b2166650122b7a51fcb83638fa18668"}
Oct 30 13:25:12.603809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.603460    4291 scope.go:122] "RemoveContainer" containerID="4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241"
Oct 30 13:25:12.603809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.603570    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"
Oct 30 13:25:12.605447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.605426    4291 generic.go:356] "Generic (PLEG): container finished" podID="7a766dbd-6979-47b9-b75c-2385a43066c9" containerID="ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc" exitCode=0
Oct 30 13:25:12.605553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.605468    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk" event={"ID":"7a766dbd-6979-47b9-b75c-2385a43066c9","Type":"ContainerDied","Data":"ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc"}
Oct 30 13:25:12.605553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.605498    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk" event={"ID":"7a766dbd-6979-47b9-b75c-2385a43066c9","Type":"ContainerDied","Data":"fe84146ff67e7c9be40fa40f7e8a8d32f4885e786a20e7cc1130771d558f241c"}
Oct 30 13:25:12.605553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.605548    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"
Oct 30 13:25:12.615490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.615466    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-kube-api-access-z64vk\" (UniqueName: \"kubernetes.io/projected/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-kube-api-access-z64vk\") pod \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") "
Oct 30 13:25:12.615644 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.615617    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a766dbd-6979-47b9-b75c-2385a43066c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a766dbd-6979-47b9-b75c-2385a43066c9-plugins-config-volume\") pod \"7a766dbd-6979-47b9-b75c-2385a43066c9\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") "
Oct 30 13:25:12.615754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.615738    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a766dbd-6979-47b9-b75c-2385a43066c9-kube-api-access-ck95k\" (UniqueName: \"kubernetes.io/projected/7a766dbd-6979-47b9-b75c-2385a43066c9-kube-api-access-ck95k\") pod \"7a766dbd-6979-47b9-b75c-2385a43066c9\" (UID: \"7a766dbd-6979-47b9-b75c-2385a43066c9\") "
Oct 30 13:25:12.616013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.615995    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-plugins-config-volume\") pod \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\" (UID: \"ca49c6f6-faa0-4a22-ad62-a552ec4484d5\") "
Oct 30 13:25:12.616354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.616321    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a766dbd-6979-47b9-b75c-2385a43066c9-plugins-config-volume" pod "7a766dbd-6979-47b9-b75c-2385a43066c9" (UID: "7a766dbd-6979-47b9-b75c-2385a43066c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:12.616760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.616740    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7a766dbd-6979-47b9-b75c-2385a43066c9-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:12.616760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.616748    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-plugins-config-volume" pod "ca49c6f6-faa0-4a22-ad62-a552ec4484d5" (UID: "ca49c6f6-faa0-4a22-ad62-a552ec4484d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:12.617109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.617070    4291 scope.go:122] "RemoveContainer" containerID="4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241"
Oct 30 13:25:12.618042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.618015    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-kube-api-access-z64vk" pod "ca49c6f6-faa0-4a22-ad62-a552ec4484d5" (UID: "ca49c6f6-faa0-4a22-ad62-a552ec4484d5"). InnerVolumeSpecName "kube-api-access-z64vk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:12.618444 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:25:12.618421    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241\": container with ID starting with 4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241 not found: ID does not exist" containerID="4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241"
Oct 30 13:25:12.618561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.618543    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241"} err="failed to get container status \"4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241\": rpc error: code = NotFound desc = could not find container \"4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241\": container with ID starting with 4a8801a9a8c544f104df37af1bdca46db449c6fd285669cbf30d7b2867b80241 not found: ID does not exist"
Oct 30 13:25:12.618646 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.618624    4291 scope.go:122] "RemoveContainer" containerID="ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc"
Oct 30 13:25:12.619050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.619030    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a766dbd-6979-47b9-b75c-2385a43066c9-kube-api-access-ck95k" pod "7a766dbd-6979-47b9-b75c-2385a43066c9" (UID: "7a766dbd-6979-47b9-b75c-2385a43066c9"). InnerVolumeSpecName "kube-api-access-ck95k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:12.628056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.628040    4291 scope.go:122] "RemoveContainer" containerID="ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc"
Oct 30 13:25:12.628366 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:25:12.628344    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc\": container with ID starting with ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc not found: ID does not exist" containerID="ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc"
Oct 30 13:25:12.628425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.628379    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc"} err="failed to get container status \"ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc\": rpc error: code = NotFound desc = could not find container \"ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc\": container with ID starting with ab01b7658070db3e5b3041a2e0c61de3d80889ee2ef0ce2ce24d1eec5507a7bc not found: ID does not exist"
Oct 30 13:25:12.707831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.707742    4291 patch_prober.go:29] interesting pod/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:25:12.707831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.707798    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:25:12.717494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.717464    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:12.717494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.717493    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z64vk\" (UniqueName: \"kubernetes.io/projected/ca49c6f6-faa0-4a22-ad62-a552ec4484d5-kube-api-access-z64vk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:12.717667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.717503    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ck95k\" (UniqueName: \"kubernetes.io/projected/7a766dbd-6979-47b9-b75c-2385a43066c9-kube-api-access-ck95k\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:12.977952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.977866    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"]
Oct 30 13:25:12.984304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:12.984269    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-8fb1a8-84c7f5b7d8-lfgmk"]
Oct 30 13:25:13.003803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:13.003766    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"]
Oct 30 13:25:13.006949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:13.006918    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-8fb1a8-8b589d679-xtcjr"]
Oct 30 13:25:13.707722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:13.707678    4291 patch_prober.go:29] interesting pod/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:25:13.708128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:13.707745    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:25:13.872660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:13.865741    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:25:13.884550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:13.884518    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a766dbd-6979-47b9-b75c-2385a43066c9" path="/var/lib/kubelet/pods/7a766dbd-6979-47b9-b75c-2385a43066c9/volumes"
Oct 30 13:25:13.885002 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:13.884980    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca49c6f6-faa0-4a22-ad62-a552ec4484d5" path="/var/lib/kubelet/pods/ca49c6f6-faa0-4a22-ad62-a552ec4484d5/volumes"
Oct 30 13:25:14.707264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:14.707224    4291 patch_prober.go:29] interesting pod/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused" start-of-body=
Oct 30 13:25:14.707475 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:14.707306    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.122:15021/healthz/ready\": dial tcp 10.133.0.122:15021: connect: connection refused"
Oct 30 13:25:15.769664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:15.769262    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:25:15.772532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:15.772504    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:25:22.534023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:22.533982    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:25:25.098698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.098657    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"]
Oct 30 13:25:25.099480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099460    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7a766dbd-6979-47b9-b75c-2385a43066c9" containerName="payload-processing"
Oct 30 13:25:25.099555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099482    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a766dbd-6979-47b9-b75c-2385a43066c9" containerName="payload-processing"
Oct 30 13:25:25.099555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099497    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ca49c6f6-faa0-4a22-ad62-a552ec4484d5" containerName="payload-pre-processing"
Oct 30 13:25:25.099555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099506    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca49c6f6-faa0-4a22-ad62-a552ec4484d5" containerName="payload-pre-processing"
Oct 30 13:25:25.099735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099568    4291 cpu_manager.go:414] "containerMap: removing container" podUID="81a3aed1-f1e2-4d11-91c6-61a4d930d40b" containerName="revoke-keys"
Oct 30 13:25:25.099735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099576    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81a3aed1-f1e2-4d11-91c6-61a4d930d40b" containerName="revoke-keys"
Oct 30 13:25:25.099845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099735    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="81a3aed1-f1e2-4d11-91c6-61a4d930d40b" containerName="revoke-keys"
Oct 30 13:25:25.099845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099751    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a766dbd-6979-47b9-b75c-2385a43066c9" containerName="payload-processing"
Oct 30 13:25:25.099845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.099760    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca49c6f6-faa0-4a22-ad62-a552ec4484d5" containerName="payload-pre-processing"
Oct 30 13:25:25.105991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.105968    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.121227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.120958    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:25:25.129230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.129191    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"]
Oct 30 13:25:25.129982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.129958    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:25:25.130155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.129978    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:25:25.204769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.204734    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.204982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.204870    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzbbv\" (UniqueName: \"kubernetes.io/projected/abd7a72e-185d-4d33-a64c-d40b2edf97e3-kube-api-access-gzbbv\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.205043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.205018    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.220420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.220388    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:25.306203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.306166    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.306403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.306282    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.306403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.306349    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzbbv\" (UniqueName: \"kubernetes.io/projected/abd7a72e-185d-4d33-a64c-d40b2edf97e3-kube-api-access-gzbbv\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.307061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.307034    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.308766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.308739    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.325418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.325391    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzbbv\" (UniqueName: \"kubernetes.io/projected/abd7a72e-185d-4d33-a64c-d40b2edf97e3-kube-api-access-gzbbv\") pod \"maas-api-revoke-sub-0b651407-w2n5g\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.443539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.443447    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:25.874135 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:25:25.874104    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd7a72e_185d_4d33_a64c_d40b2edf97e3.slice/crio-8f9ed82e2a2ae9bf835fc0c5571e1405db154dbfc06954820df4dc76ad441859 WatchSource:0}: Error finding container 8f9ed82e2a2ae9bf835fc0c5571e1405db154dbfc06954820df4dc76ad441859: Status 404 returned error can't find the container with id 8f9ed82e2a2ae9bf835fc0c5571e1405db154dbfc06954820df4dc76ad441859
Oct 30 13:25:25.882108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:25.882084    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"]
Oct 30 13:25:26.018957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:26.018920    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"]
Oct 30 13:25:26.023077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:26.023050    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c9e946-m55b7"]
Oct 30 13:25:26.729494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:26.729463    4291 generic.go:356] "Generic (PLEG): container finished" podID="abd7a72e-185d-4d33-a64c-d40b2edf97e3" containerID="b621c1cbfaec9117542e0a450d95934bef94a8494aa644b92cc78cc41e791f12" exitCode=0
Oct 30 13:25:26.729909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:26.729539    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g" event={"ID":"abd7a72e-185d-4d33-a64c-d40b2edf97e3","Type":"ContainerDied","Data":"b621c1cbfaec9117542e0a450d95934bef94a8494aa644b92cc78cc41e791f12"}
Oct 30 13:25:26.729909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:26.729619    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g" event={"ID":"abd7a72e-185d-4d33-a64c-d40b2edf97e3","Type":"ContainerStarted","Data":"8f9ed82e2a2ae9bf835fc0c5571e1405db154dbfc06954820df4dc76ad441859"}
Oct 30 13:25:27.889264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.889222    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6325be5-9c9c-499a-a8c3-ab4850624fd9" path="/var/lib/kubelet/pods/d6325be5-9c9c-499a-a8c3-ab4850624fd9/volumes"
Oct 30 13:25:27.934138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.934096    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:27.980687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.980509    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-xks-ca\") pod \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") "
Oct 30 13:25:27.981919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.981887    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-openshift-ca\") pod \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") "
Oct 30 13:25:27.989541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.983584    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd7a72e-185d-4d33-a64c-d40b2edf97e3-kube-api-access-gzbbv\" (UniqueName: \"kubernetes.io/projected/abd7a72e-185d-4d33-a64c-d40b2edf97e3-kube-api-access-gzbbv\") pod \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\" (UID: \"abd7a72e-185d-4d33-a64c-d40b2edf97e3\") "
Oct 30 13:25:27.990077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.986822    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-openshift-ca" pod "abd7a72e-185d-4d33-a64c-d40b2edf97e3" (UID: "abd7a72e-185d-4d33-a64c-d40b2edf97e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:27.990197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.987391    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd7a72e-185d-4d33-a64c-d40b2edf97e3-kube-api-access-gzbbv" pod "abd7a72e-185d-4d33-a64c-d40b2edf97e3" (UID: "abd7a72e-185d-4d33-a64c-d40b2edf97e3"). InnerVolumeSpecName "kube-api-access-gzbbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:27.990331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.990293    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-xks-ca" pod "abd7a72e-185d-4d33-a64c-d40b2edf97e3" (UID: "abd7a72e-185d-4d33-a64c-d40b2edf97e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:27.991293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.991017    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:27.991293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.991046    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzbbv\" (UniqueName: \"kubernetes.io/projected/abd7a72e-185d-4d33-a64c-d40b2edf97e3-kube-api-access-gzbbv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:27.991293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:27.991064    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd7a72e-185d-4d33-a64c-d40b2edf97e3-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:28.744131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:28.744086    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g" event={"ID":"abd7a72e-185d-4d33-a64c-d40b2edf97e3","Type":"ContainerDied","Data":"8f9ed82e2a2ae9bf835fc0c5571e1405db154dbfc06954820df4dc76ad441859"}
Oct 30 13:25:28.744131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:28.744136    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f9ed82e2a2ae9bf835fc0c5571e1405db154dbfc06954820df4dc76ad441859"
Oct 30 13:25:28.744343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:28.744101    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"
Oct 30 13:25:30.017101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:30.017065    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"]
Oct 30 13:25:30.020151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:30.020129    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-3a82db-7tp4d"]
Oct 30 13:25:31.880538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:31.880498    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="682ccddd-dcf2-4cba-a805-669cb2cfcf52" path="/var/lib/kubelet/pods/682ccddd-dcf2-4cba-a805-669cb2cfcf52/volumes"
Oct 30 13:25:34.279432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.279396    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"]
Oct 30 13:25:34.279949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.279935    4291 cpu_manager.go:414] "containerMap: removing container" podUID="abd7a72e-185d-4d33-a64c-d40b2edf97e3" containerName="revoke-keys"
Oct 30 13:25:34.279996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.279950    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd7a72e-185d-4d33-a64c-d40b2edf97e3" containerName="revoke-keys"
Oct 30 13:25:34.280081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.280073    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd7a72e-185d-4d33-a64c-d40b2edf97e3" containerName="revoke-keys"
Oct 30 13:25:34.283839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.283813    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.300009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.299978    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"]
Oct 30 13:25:34.308654 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.308605    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:34.308866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.308625    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:25:34.308866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.308695    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:25:34.312450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.312425    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:25:34.390102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.390059    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpwbv\" (UniqueName: \"kubernetes.io/projected/fd02cdc5-cb74-4957-bd6e-f679407d9070-kube-api-access-dpwbv\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.390311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.390110    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.390411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.390381    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.491290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.491249    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.491500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.491402    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpwbv\" (UniqueName: \"kubernetes.io/projected/fd02cdc5-cb74-4957-bd6e-f679407d9070-kube-api-access-dpwbv\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.491500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.491433    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.492077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.492049    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.494034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.494010    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.511098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.511066    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpwbv\" (UniqueName: \"kubernetes.io/projected/fd02cdc5-cb74-4957-bd6e-f679407d9070-kube-api-access-dpwbv\") pod \"maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.598593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.598551    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:34.904890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:34.904860    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"]
Oct 30 13:25:34.905794 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:25:34.905770    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd02cdc5_cb74_4957_bd6e_f679407d9070.slice/crio-5607806abce9a42b9c9f672c0013cbd417373676f38909a43258943f9b918bd1 WatchSource:0}: Error finding container 5607806abce9a42b9c9f672c0013cbd417373676f38909a43258943f9b918bd1: Status 404 returned error can't find the container with id 5607806abce9a42b9c9f672c0013cbd417373676f38909a43258943f9b918bd1
Oct 30 13:25:35.862003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:35.861971    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd02cdc5-cb74-4957-bd6e-f679407d9070" containerID="ff55d5d706ca10ad1702bbac1e790f6f0a31fa3fdd8a1d23aa906383be21197e" exitCode=0
Oct 30 13:25:35.862394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:35.862053    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s" event={"ID":"fd02cdc5-cb74-4957-bd6e-f679407d9070","Type":"ContainerDied","Data":"ff55d5d706ca10ad1702bbac1e790f6f0a31fa3fdd8a1d23aa906383be21197e"}
Oct 30 13:25:35.862394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:35.862106    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s" event={"ID":"fd02cdc5-cb74-4957-bd6e-f679407d9070","Type":"ContainerStarted","Data":"5607806abce9a42b9c9f672c0013cbd417373676f38909a43258943f9b918bd1"}
Oct 30 13:25:37.035975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.035950    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:37.154827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.154738    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd02cdc5-cb74-4957-bd6e-f679407d9070-kube-api-access-dpwbv\" (UniqueName: \"kubernetes.io/projected/fd02cdc5-cb74-4957-bd6e-f679407d9070-kube-api-access-dpwbv\") pod \"fd02cdc5-cb74-4957-bd6e-f679407d9070\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") "
Oct 30 13:25:37.154827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.154806    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-openshift-ca\") pod \"fd02cdc5-cb74-4957-bd6e-f679407d9070\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") "
Oct 30 13:25:37.155062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.154895    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-xks-ca\") pod \"fd02cdc5-cb74-4957-bd6e-f679407d9070\" (UID: \"fd02cdc5-cb74-4957-bd6e-f679407d9070\") "
Oct 30 13:25:37.155242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.155213    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-openshift-ca" pod "fd02cdc5-cb74-4957-bd6e-f679407d9070" (UID: "fd02cdc5-cb74-4957-bd6e-f679407d9070"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:37.156977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.156952    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-xks-ca" pod "fd02cdc5-cb74-4957-bd6e-f679407d9070" (UID: "fd02cdc5-cb74-4957-bd6e-f679407d9070"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:25:37.157075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.157032    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd02cdc5-cb74-4957-bd6e-f679407d9070-kube-api-access-dpwbv" pod "fd02cdc5-cb74-4957-bd6e-f679407d9070" (UID: "fd02cdc5-cb74-4957-bd6e-f679407d9070"). InnerVolumeSpecName "kube-api-access-dpwbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:37.256370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.256338    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:37.256370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.256369    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd02cdc5-cb74-4957-bd6e-f679407d9070-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:37.256370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.256379    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpwbv\" (UniqueName: \"kubernetes.io/projected/fd02cdc5-cb74-4957-bd6e-f679407d9070-kube-api-access-dpwbv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:37.877701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.877671    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"
Oct 30 13:25:37.881028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.881001    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s" event={"ID":"fd02cdc5-cb74-4957-bd6e-f679407d9070","Type":"ContainerDied","Data":"5607806abce9a42b9c9f672c0013cbd417373676f38909a43258943f9b918bd1"}
Oct 30 13:25:37.881177 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:37.881046    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5607806abce9a42b9c9f672c0013cbd417373676f38909a43258943f9b918bd1"
Oct 30 13:25:38.210871 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:25:38.210789    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:25:44.701757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:44.701717    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"]
Oct 30 13:25:44.717568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:44.717532    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-8fb1a8-75lsw"]
Oct 30 13:25:44.881300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:44.881261    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"]
Oct 30 13:25:44.911385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:44.911351    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"]
Oct 30 13:25:45.032264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:45.032186    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"]
Oct 30 13:25:45.035248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:45.035220    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5940752f-jpc87"]
Oct 30 13:25:45.879560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:45.879525    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89b475c1-33e5-46e3-98c4-69bdaaabde56" path="/var/lib/kubelet/pods/89b475c1-33e5-46e3-98c4-69bdaaabde56/volumes"
Oct 30 13:25:45.880074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:45.880021    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5310cca-244f-4cba-a62c-1eef3ee28420" path="/var/lib/kubelet/pods/c5310cca-244f-4cba-a62c-1eef3ee28420/volumes"
Oct 30 13:25:46.504085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.504050    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"]
Oct 30 13:25:46.504295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.504274    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l" podUID="21a22c44-e47f-43a3-ac45-90ff7853e5ba" containerName="payload-processing" containerID="cri-o://d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12" gracePeriod=30
Oct 30 13:25:46.801727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.801700    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:25:46.921092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.913060    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a22c44-e47f-43a3-ac45-90ff7853e5ba-kube-api-access-5sppp\" (UniqueName: \"kubernetes.io/projected/21a22c44-e47f-43a3-ac45-90ff7853e5ba-kube-api-access-5sppp\") pod \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") "
Oct 30 13:25:46.921092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.913251    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a22c44-e47f-43a3-ac45-90ff7853e5ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a22c44-e47f-43a3-ac45-90ff7853e5ba-plugins-config-volume\") pod \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\" (UID: \"21a22c44-e47f-43a3-ac45-90ff7853e5ba\") "
Oct 30 13:25:46.926385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.925873    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a22c44-e47f-43a3-ac45-90ff7853e5ba-plugins-config-volume" pod "21a22c44-e47f-43a3-ac45-90ff7853e5ba" (UID: "21a22c44-e47f-43a3-ac45-90ff7853e5ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:46.926385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.926346    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21a22c44-e47f-43a3-ac45-90ff7853e5ba-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:46.948915 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:46.948850    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a22c44-e47f-43a3-ac45-90ff7853e5ba-kube-api-access-5sppp" pod "21a22c44-e47f-43a3-ac45-90ff7853e5ba" (UID: "21a22c44-e47f-43a3-ac45-90ff7853e5ba"). InnerVolumeSpecName "kube-api-access-5sppp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:47.030149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.028075    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5sppp\" (UniqueName: \"kubernetes.io/projected/21a22c44-e47f-43a3-ac45-90ff7853e5ba-kube-api-access-5sppp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:47.036538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.036510    4291 generic.go:356] "Generic (PLEG): container finished" podID="21a22c44-e47f-43a3-ac45-90ff7853e5ba" containerID="d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12" exitCode=0
Oct 30 13:25:47.036906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.036791    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"
Oct 30 13:25:47.037609 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.036790    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l" event={"ID":"21a22c44-e47f-43a3-ac45-90ff7853e5ba","Type":"ContainerDied","Data":"d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12"}
Oct 30 13:25:47.037966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.037775    4291 scope.go:122] "RemoveContainer" containerID="d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12"
Oct 30 13:25:47.046714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.046674    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l" event={"ID":"21a22c44-e47f-43a3-ac45-90ff7853e5ba","Type":"ContainerDied","Data":"4927e1cae3634c6ae44e243c514b9716e442f439566f2cab579e7c8ce62693c8"}
Oct 30 13:25:47.080598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.080574    4291 scope.go:122] "RemoveContainer" containerID="d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12"
Oct 30 13:25:47.080927 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:25:47.080907    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12\": container with ID starting with d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12 not found: ID does not exist" containerID="d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12"
Oct 30 13:25:47.081029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.080935    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12"} err="failed to get container status \"d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12\": rpc error: code = NotFound desc = could not find container \"d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12\": container with ID starting with d1112fdd39ebdae7e80851ce8c6946b5a1dc9bf5cfe259de4b94985a5fee8e12 not found: ID does not exist"
Oct 30 13:25:47.119156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.119124    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"]
Oct 30 13:25:47.124737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.124705    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-8ac28e-54cf8969b4-2jf5l"]
Oct 30 13:25:47.332284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.332202    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"]
Oct 30 13:25:47.334957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.334933    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-8ac28e-6l42c"]
Oct 30 13:25:47.878418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.878388    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a22c44-e47f-43a3-ac45-90ff7853e5ba" path="/var/lib/kubelet/pods/21a22c44-e47f-43a3-ac45-90ff7853e5ba/volumes"
Oct 30 13:25:47.878854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:47.878840    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81a3aed1-f1e2-4d11-91c6-61a4d930d40b" path="/var/lib/kubelet/pods/81a3aed1-f1e2-4d11-91c6-61a4d930d40b/volumes"
Oct 30 13:25:49.882021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:49.881971    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr" podUID="86d7d27a-af57-46ef-be85-409ed651042c" containerName="payload-processing" containerID="cri-o://a427493084ea16af6c0f5c09f95f0892b5d4621a6d26081d25a7689cc1c1141f" gracePeriod=25
Oct 30 13:25:49.911911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:49.911874    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p" podUID="53cfe0b4-bc28-4e95-966d-840c3167da56" containerName="payload-pre-processing" containerID="cri-o://49b24f8d6fe42e696322e53d0d4804c9953d93efe2697bb0fad250e878650997" gracePeriod=25
Oct 30 13:25:50.105225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.105174    4291 generic.go:356] "Generic (PLEG): container finished" podID="86d7d27a-af57-46ef-be85-409ed651042c" containerID="a427493084ea16af6c0f5c09f95f0892b5d4621a6d26081d25a7689cc1c1141f" exitCode=0
Oct 30 13:25:50.105426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.105242    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr" event={"ID":"86d7d27a-af57-46ef-be85-409ed651042c","Type":"ContainerDied","Data":"a427493084ea16af6c0f5c09f95f0892b5d4621a6d26081d25a7689cc1c1141f"}
Oct 30 13:25:50.107715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.107686    4291 generic.go:356] "Generic (PLEG): container finished" podID="53cfe0b4-bc28-4e95-966d-840c3167da56" containerID="49b24f8d6fe42e696322e53d0d4804c9953d93efe2697bb0fad250e878650997" exitCode=0
Oct 30 13:25:50.107946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.107926    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p" event={"ID":"53cfe0b4-bc28-4e95-966d-840c3167da56","Type":"ContainerDied","Data":"49b24f8d6fe42e696322e53d0d4804c9953d93efe2697bb0fad250e878650997"}
Oct 30 13:25:50.176508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.176465    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:25:50.251905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.251867    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:25:50.275359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.275329    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86d7d27a-af57-46ef-be85-409ed651042c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7d27a-af57-46ef-be85-409ed651042c-plugins-config-volume\") pod \"86d7d27a-af57-46ef-be85-409ed651042c\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") "
Oct 30 13:25:50.275531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.275370    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86d7d27a-af57-46ef-be85-409ed651042c-kube-api-access-tlbk4\" (UniqueName: \"kubernetes.io/projected/86d7d27a-af57-46ef-be85-409ed651042c-kube-api-access-tlbk4\") pod \"86d7d27a-af57-46ef-be85-409ed651042c\" (UID: \"86d7d27a-af57-46ef-be85-409ed651042c\") "
Oct 30 13:25:50.275531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.275455    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cfe0b4-bc28-4e95-966d-840c3167da56-kube-api-access-gw29q\" (UniqueName: \"kubernetes.io/projected/53cfe0b4-bc28-4e95-966d-840c3167da56-kube-api-access-gw29q\") pod \"53cfe0b4-bc28-4e95-966d-840c3167da56\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") "
Oct 30 13:25:50.275685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.275541    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cfe0b4-bc28-4e95-966d-840c3167da56-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cfe0b4-bc28-4e95-966d-840c3167da56-plugins-config-volume\") pod \"53cfe0b4-bc28-4e95-966d-840c3167da56\" (UID: \"53cfe0b4-bc28-4e95-966d-840c3167da56\") "
Oct 30 13:25:50.275753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.275719    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86d7d27a-af57-46ef-be85-409ed651042c-plugins-config-volume" pod "86d7d27a-af57-46ef-be85-409ed651042c" (UID: "86d7d27a-af57-46ef-be85-409ed651042c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:50.275990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.275967    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cfe0b4-bc28-4e95-966d-840c3167da56-plugins-config-volume" pod "53cfe0b4-bc28-4e95-966d-840c3167da56" (UID: "53cfe0b4-bc28-4e95-966d-840c3167da56"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:25:50.276092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.276051    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86d7d27a-af57-46ef-be85-409ed651042c-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:50.276092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.276063    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53cfe0b4-bc28-4e95-966d-840c3167da56-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:50.277489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.277469    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86d7d27a-af57-46ef-be85-409ed651042c-kube-api-access-tlbk4" pod "86d7d27a-af57-46ef-be85-409ed651042c" (UID: "86d7d27a-af57-46ef-be85-409ed651042c"). InnerVolumeSpecName "kube-api-access-tlbk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:50.277579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.277529    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cfe0b4-bc28-4e95-966d-840c3167da56-kube-api-access-gw29q" pod "53cfe0b4-bc28-4e95-966d-840c3167da56" (UID: "53cfe0b4-bc28-4e95-966d-840c3167da56"). InnerVolumeSpecName "kube-api-access-gw29q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:25:50.377503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.377463    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlbk4\" (UniqueName: \"kubernetes.io/projected/86d7d27a-af57-46ef-be85-409ed651042c-kube-api-access-tlbk4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:50.377503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:50.377495    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw29q\" (UniqueName: \"kubernetes.io/projected/53cfe0b4-bc28-4e95-966d-840c3167da56-kube-api-access-gw29q\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:25:51.116116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.116085    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"
Oct 30 13:25:51.116623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.116108    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p" event={"ID":"53cfe0b4-bc28-4e95-966d-840c3167da56","Type":"ContainerDied","Data":"9170e7bee0ac9864ab35efcb990de968897b62abd2dcf24a324814293f36d552"}
Oct 30 13:25:51.116623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.116188    4291 scope.go:122] "RemoveContainer" containerID="49b24f8d6fe42e696322e53d0d4804c9953d93efe2697bb0fad250e878650997"
Oct 30 13:25:51.117963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.117934    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr" event={"ID":"86d7d27a-af57-46ef-be85-409ed651042c","Type":"ContainerDied","Data":"8ba4f5e30d2bdc755ee29abe1a8ab767150cf6a437cb4ee6c319b7d9316e9ce9"}
Oct 30 13:25:51.118055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.117999    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"
Oct 30 13:25:51.127263 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.127245    4291 scope.go:122] "RemoveContainer" containerID="a427493084ea16af6c0f5c09f95f0892b5d4621a6d26081d25a7689cc1c1141f"
Oct 30 13:25:51.166531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.166462    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"]
Oct 30 13:25:51.169850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.169830    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-e3e617-6d79bb45fd-6cmsr"]
Oct 30 13:25:51.203215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.203186    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"]
Oct 30 13:25:51.206075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.206036    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e3e617-69f5684df9-cx64p"]
Oct 30 13:25:51.879500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.879467    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cfe0b4-bc28-4e95-966d-840c3167da56" path="/var/lib/kubelet/pods/53cfe0b4-bc28-4e95-966d-840c3167da56/volumes"
Oct 30 13:25:51.879952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:51.879938    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86d7d27a-af57-46ef-be85-409ed651042c" path="/var/lib/kubelet/pods/86d7d27a-af57-46ef-be85-409ed651042c/volumes"
Oct 30 13:25:57.426859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.426820    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"]
Oct 30 13:25:57.427661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427618    4291 cpu_manager.go:414] "containerMap: removing container" podUID="86d7d27a-af57-46ef-be85-409ed651042c" containerName="payload-processing"
Oct 30 13:25:57.427661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427653    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86d7d27a-af57-46ef-be85-409ed651042c" containerName="payload-processing"
Oct 30 13:25:57.427769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427702    4291 cpu_manager.go:414] "containerMap: removing container" podUID="53cfe0b4-bc28-4e95-966d-840c3167da56" containerName="payload-pre-processing"
Oct 30 13:25:57.427769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427712    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cfe0b4-bc28-4e95-966d-840c3167da56" containerName="payload-pre-processing"
Oct 30 13:25:57.427769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427736    4291 cpu_manager.go:414] "containerMap: removing container" podUID="21a22c44-e47f-43a3-ac45-90ff7853e5ba" containerName="payload-processing"
Oct 30 13:25:57.427769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427745    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a22c44-e47f-43a3-ac45-90ff7853e5ba" containerName="payload-processing"
Oct 30 13:25:57.427769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427759    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd02cdc5-cb74-4957-bd6e-f679407d9070" containerName="revoke-keys"
Oct 30 13:25:57.427769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427768    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd02cdc5-cb74-4957-bd6e-f679407d9070" containerName="revoke-keys"
Oct 30 13:25:57.428005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427902    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cfe0b4-bc28-4e95-966d-840c3167da56" containerName="payload-pre-processing"
Oct 30 13:25:57.428005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427920    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a22c44-e47f-43a3-ac45-90ff7853e5ba" containerName="payload-processing"
Oct 30 13:25:57.428005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427931    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd02cdc5-cb74-4957-bd6e-f679407d9070" containerName="revoke-keys"
Oct 30 13:25:57.428005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.427945    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="86d7d27a-af57-46ef-be85-409ed651042c" containerName="payload-processing"
Oct 30 13:25:57.433346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.433323    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.447814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.447781    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"]
Oct 30 13:25:57.454616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.454584    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:25:57.455241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.454672    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:25:57.455241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.455191    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:25:57.459031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.459004    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:25:57.585713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.585678    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.585713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.585717    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.585930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.585877    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbvk\" (UniqueName: \"kubernetes.io/projected/29c90f38-8cb2-4158-9535-e276c3b8703b-kube-api-access-9sbvk\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.687798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.687716    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.687798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.687763    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.688027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.687870    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbvk\" (UniqueName: \"kubernetes.io/projected/29c90f38-8cb2-4158-9535-e276c3b8703b-kube-api-access-9sbvk\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.688443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.688418    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.690034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.690012    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.709880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.709847    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbvk\" (UniqueName: \"kubernetes.io/projected/29c90f38-8cb2-4158-9535-e276c3b8703b-kube-api-access-9sbvk\") pod \"maas-api-revoke-sub-3aa8dd3b-dmqzw\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:57.751014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:57.750984    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:25:58.055871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:58.055843    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"]
Oct 30 13:25:58.057890 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:25:58.057850    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29c90f38_8cb2_4158_9535_e276c3b8703b.slice/crio-e41e1d112eec4a2def8079a6855bb33c0a359c8c236296da3a773877ecee92b2 WatchSource:0}: Error finding container e41e1d112eec4a2def8079a6855bb33c0a359c8c236296da3a773877ecee92b2: Status 404 returned error can't find the container with id e41e1d112eec4a2def8079a6855bb33c0a359c8c236296da3a773877ecee92b2
Oct 30 13:25:58.168665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:58.168617    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw" event={"ID":"29c90f38-8cb2-4158-9535-e276c3b8703b","Type":"ContainerStarted","Data":"e41e1d112eec4a2def8079a6855bb33c0a359c8c236296da3a773877ecee92b2"}
Oct 30 13:25:59.176431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:59.176343    4291 generic.go:356] "Generic (PLEG): container finished" podID="29c90f38-8cb2-4158-9535-e276c3b8703b" containerID="36bac5979a1ffe7e4e0ced81102ec4ae5e8bd558b293796c4cebde54a285bb9d" exitCode=0
Oct 30 13:25:59.176832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:25:59.176431    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw" event={"ID":"29c90f38-8cb2-4158-9535-e276c3b8703b","Type":"ContainerDied","Data":"36bac5979a1ffe7e4e0ced81102ec4ae5e8bd558b293796c4cebde54a285bb9d"}
Oct 30 13:26:00.349263 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.349237    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:26:00.425835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.425788    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29c90f38-8cb2-4158-9535-e276c3b8703b-kube-api-access-9sbvk\" (UniqueName: \"kubernetes.io/projected/29c90f38-8cb2-4158-9535-e276c3b8703b-kube-api-access-9sbvk\") pod \"29c90f38-8cb2-4158-9535-e276c3b8703b\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") "
Oct 30 13:26:00.425835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.425831    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-openshift-ca\") pod \"29c90f38-8cb2-4158-9535-e276c3b8703b\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") "
Oct 30 13:26:00.426214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.426191    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-xks-ca\") pod \"29c90f38-8cb2-4158-9535-e276c3b8703b\" (UID: \"29c90f38-8cb2-4158-9535-e276c3b8703b\") "
Oct 30 13:26:00.426337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.426307    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-openshift-ca" pod "29c90f38-8cb2-4158-9535-e276c3b8703b" (UID: "29c90f38-8cb2-4158-9535-e276c3b8703b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:00.426828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.426800    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:00.428236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.428202    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c90f38-8cb2-4158-9535-e276c3b8703b-kube-api-access-9sbvk" pod "29c90f38-8cb2-4158-9535-e276c3b8703b" (UID: "29c90f38-8cb2-4158-9535-e276c3b8703b"). InnerVolumeSpecName "kube-api-access-9sbvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:00.428700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.428678    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-xks-ca" pod "29c90f38-8cb2-4158-9535-e276c3b8703b" (UID: "29c90f38-8cb2-4158-9535-e276c3b8703b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:00.528614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.528578    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29c90f38-8cb2-4158-9535-e276c3b8703b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:00.528614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:00.528609    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbvk\" (UniqueName: \"kubernetes.io/projected/29c90f38-8cb2-4158-9535-e276c3b8703b-kube-api-access-9sbvk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:01.066398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:01.064270    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:26:01.197015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:01.196991    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"
Oct 30 13:26:01.197163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:01.197024    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw" event={"ID":"29c90f38-8cb2-4158-9535-e276c3b8703b","Type":"ContainerDied","Data":"e41e1d112eec4a2def8079a6855bb33c0a359c8c236296da3a773877ecee92b2"}
Oct 30 13:26:01.197163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:01.197071    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e41e1d112eec4a2def8079a6855bb33c0a359c8c236296da3a773877ecee92b2"
Oct 30 13:26:08.210152 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:26:08.210095    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:26:10.915889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:10.915856    4291 scope.go:122] "RemoveContainer" containerID="7eaef2a957cd05b8723a306590a789cc852c2c70176eb6c20d8e2c664e1980b4"
Oct 30 13:26:10.925767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:10.925743    4291 scope.go:122] "RemoveContainer" containerID="bce45a69d907e6b2694353c4df369045a6cc9b223eb31dee683583d1b27e136b"
Oct 30 13:26:10.935219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:10.935197    4291 scope.go:122] "RemoveContainer" containerID="04687bb05526c06a912ce34ff70a2fd443ffe0063ea6d83f73bdda40389006a1"
Oct 30 13:26:10.944236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:10.944216    4291 scope.go:122] "RemoveContainer" containerID="e099c2ee51aab1e4410e0cff3443962ecadf4933ff5ee381f346f16c0cfc56fe"
Oct 30 13:26:13.991976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:13.991940    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"]
Oct 30 13:26:14.003733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.003701    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e3e617-2vs4s"]
Oct 30 13:26:14.422496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.422462    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"]
Oct 30 13:26:14.423007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.422993    4291 cpu_manager.go:414] "containerMap: removing container" podUID="29c90f38-8cb2-4158-9535-e276c3b8703b" containerName="revoke-keys"
Oct 30 13:26:14.423064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.423008    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29c90f38-8cb2-4158-9535-e276c3b8703b" containerName="revoke-keys"
Oct 30 13:26:14.423100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.423083    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="29c90f38-8cb2-4158-9535-e276c3b8703b" containerName="revoke-keys"
Oct 30 13:26:14.428187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.428168    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.443055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.443029    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"]
Oct 30 13:26:14.480447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.480391    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:14.480657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.480530    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:26:14.524270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.524238    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:26:14.524270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.524284    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:26:14.553924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.553892    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.554092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.554073    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.554156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.554110    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c4sk\" (UniqueName: \"kubernetes.io/projected/957f86af-96e7-4bc6-b534-e1e166bf8392-kube-api-access-8c4sk\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.655543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.655506    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.655757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.655666    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.655757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.655700    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c4sk\" (UniqueName: \"kubernetes.io/projected/957f86af-96e7-4bc6-b534-e1e166bf8392-kube-api-access-8c4sk\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.656306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.656285    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.657877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.657860    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.674562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.674484    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c4sk\" (UniqueName: \"kubernetes.io/projected/957f86af-96e7-4bc6-b534-e1e166bf8392-kube-api-access-8c4sk\") pod \"maas-api-revoke-sub-1f8ff967-pz78l\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:14.743035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:14.742999    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:15.047982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:15.047956    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"]
Oct 30 13:26:15.050069 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:26:15.050043    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod957f86af_96e7_4bc6_b534_e1e166bf8392.slice/crio-d92f5bc78c300cc8582f3b069f28c9e3d965bc266cfd5713580a038b4a5be444 WatchSource:0}: Error finding container d92f5bc78c300cc8582f3b069f28c9e3d965bc266cfd5713580a038b4a5be444: Status 404 returned error can't find the container with id d92f5bc78c300cc8582f3b069f28c9e3d965bc266cfd5713580a038b4a5be444
Oct 30 13:26:15.308128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:15.308033    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l" event={"ID":"957f86af-96e7-4bc6-b534-e1e166bf8392","Type":"ContainerStarted","Data":"d92f5bc78c300cc8582f3b069f28c9e3d965bc266cfd5713580a038b4a5be444"}
Oct 30 13:26:15.878755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:15.878722    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd02cdc5-cb74-4957-bd6e-f679407d9070" path="/var/lib/kubelet/pods/fd02cdc5-cb74-4957-bd6e-f679407d9070/volumes"
Oct 30 13:26:16.316118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:16.316039    4291 generic.go:356] "Generic (PLEG): container finished" podID="957f86af-96e7-4bc6-b534-e1e166bf8392" containerID="655bebedf687f6dd37c237542966746ffad102302d8e5943bec2a889eb5d9a43" exitCode=0
Oct 30 13:26:16.316493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:16.316120    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l" event={"ID":"957f86af-96e7-4bc6-b534-e1e166bf8392","Type":"ContainerDied","Data":"655bebedf687f6dd37c237542966746ffad102302d8e5943bec2a889eb5d9a43"}
Oct 30 13:26:17.482028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.481993    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:17.602287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.602237    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-xks-ca\") pod \"957f86af-96e7-4bc6-b534-e1e166bf8392\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") "
Oct 30 13:26:17.602501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.602413    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-openshift-ca\") pod \"957f86af-96e7-4bc6-b534-e1e166bf8392\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") "
Oct 30 13:26:17.602501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.602476    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/957f86af-96e7-4bc6-b534-e1e166bf8392-kube-api-access-8c4sk\" (UniqueName: \"kubernetes.io/projected/957f86af-96e7-4bc6-b534-e1e166bf8392-kube-api-access-8c4sk\") pod \"957f86af-96e7-4bc6-b534-e1e166bf8392\" (UID: \"957f86af-96e7-4bc6-b534-e1e166bf8392\") "
Oct 30 13:26:17.604416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.604235    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-openshift-ca" pod "957f86af-96e7-4bc6-b534-e1e166bf8392" (UID: "957f86af-96e7-4bc6-b534-e1e166bf8392"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:17.620871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.620840    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-xks-ca" pod "957f86af-96e7-4bc6-b534-e1e166bf8392" (UID: "957f86af-96e7-4bc6-b534-e1e166bf8392"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:17.621076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.620955    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/957f86af-96e7-4bc6-b534-e1e166bf8392-kube-api-access-8c4sk" pod "957f86af-96e7-4bc6-b534-e1e166bf8392" (UID: "957f86af-96e7-4bc6-b534-e1e166bf8392"). InnerVolumeSpecName "kube-api-access-8c4sk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:17.704355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.704317    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:17.704355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.704349    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8c4sk\" (UniqueName: \"kubernetes.io/projected/957f86af-96e7-4bc6-b534-e1e166bf8392-kube-api-access-8c4sk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:17.704355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:17.704364    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/957f86af-96e7-4bc6-b534-e1e166bf8392-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:18.330622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:18.330544    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l" event={"ID":"957f86af-96e7-4bc6-b534-e1e166bf8392","Type":"ContainerDied","Data":"d92f5bc78c300cc8582f3b069f28c9e3d965bc266cfd5713580a038b4a5be444"}
Oct 30 13:26:18.330622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:18.330597    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92f5bc78c300cc8582f3b069f28c9e3d965bc266cfd5713580a038b4a5be444"
Oct 30 13:26:18.330622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:18.330557    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"
Oct 30 13:26:25.036574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:25.036471    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"]
Oct 30 13:26:25.047973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:25.047939    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20924782-8cwch"]
Oct 30 13:26:26.062660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:26.058131    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="63f29e84-8029-4d3d-8956-51937104d2d0" path="/var/lib/kubelet/pods/63f29e84-8029-4d3d-8956-51937104d2d0/volumes"
Oct 30 13:26:29.919675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.919626    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"]
Oct 30 13:26:29.920418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.920394    4291 cpu_manager.go:414] "containerMap: removing container" podUID="957f86af-96e7-4bc6-b534-e1e166bf8392" containerName="revoke-keys"
Oct 30 13:26:29.920515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.920420    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="957f86af-96e7-4bc6-b534-e1e166bf8392" containerName="revoke-keys"
Oct 30 13:26:29.920605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.920591    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="957f86af-96e7-4bc6-b534-e1e166bf8392" containerName="revoke-keys"
Oct 30 13:26:29.925125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.925103    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:29.940069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.940042    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"]
Oct 30 13:26:29.948347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.948314    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:26:29.948486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.948318    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:26:29.948486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.948429    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:29.951099 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.951070    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:26:29.997563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.997529    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:29.997811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.997712    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:29.997888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:29.997870    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4dxc\" (UniqueName: \"kubernetes.io/projected/74ab3645-5504-445f-a078-0a070972b4f2-kube-api-access-q4dxc\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.099601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.099561    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.099842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.099754    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4dxc\" (UniqueName: \"kubernetes.io/projected/74ab3645-5504-445f-a078-0a070972b4f2-kube-api-access-q4dxc\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.099911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.099854    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.100504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.100485    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.101913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.101889    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.119760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.119732    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4dxc\" (UniqueName: \"kubernetes.io/projected/74ab3645-5504-445f-a078-0a070972b4f2-kube-api-access-q4dxc\") pod \"maas-api-revoke-sub-d379a7e7-zwcqw\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.239433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.239350    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:30.559606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:30.559571    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"]
Oct 30 13:26:30.561052 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:26:30.561024    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ab3645_5504_445f_a078_0a070972b4f2.slice/crio-ed763f3a61cc349919ffd1ace89163b0ad2630afafc9f7981db7c6ea4c3ae8b0 WatchSource:0}: Error finding container ed763f3a61cc349919ffd1ace89163b0ad2630afafc9f7981db7c6ea4c3ae8b0: Status 404 returned error can't find the container with id ed763f3a61cc349919ffd1ace89163b0ad2630afafc9f7981db7c6ea4c3ae8b0
Oct 30 13:26:31.479012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:31.478981    4291 generic.go:356] "Generic (PLEG): container finished" podID="74ab3645-5504-445f-a078-0a070972b4f2" containerID="74bb4c4c6c67d6ee851c73f3cee36e9ee2ec2e4bbe6f8c92df5fc3312c894211" exitCode=0
Oct 30 13:26:31.479398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:31.479060    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw" event={"ID":"74ab3645-5504-445f-a078-0a070972b4f2","Type":"ContainerDied","Data":"74bb4c4c6c67d6ee851c73f3cee36e9ee2ec2e4bbe6f8c92df5fc3312c894211"}
Oct 30 13:26:31.479398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:31.479115    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw" event={"ID":"74ab3645-5504-445f-a078-0a070972b4f2","Type":"ContainerStarted","Data":"ed763f3a61cc349919ffd1ace89163b0ad2630afafc9f7981db7c6ea4c3ae8b0"}
Oct 30 13:26:32.644684 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.644653    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:32.738492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.738410    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-xks-ca\") pod \"74ab3645-5504-445f-a078-0a070972b4f2\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") "
Oct 30 13:26:32.738657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.738502    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-openshift-ca\") pod \"74ab3645-5504-445f-a078-0a070972b4f2\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") "
Oct 30 13:26:32.738657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.738548    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ab3645-5504-445f-a078-0a070972b4f2-kube-api-access-q4dxc\" (UniqueName: \"kubernetes.io/projected/74ab3645-5504-445f-a078-0a070972b4f2-kube-api-access-q4dxc\") pod \"74ab3645-5504-445f-a078-0a070972b4f2\" (UID: \"74ab3645-5504-445f-a078-0a070972b4f2\") "
Oct 30 13:26:32.738863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.738836    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-openshift-ca" pod "74ab3645-5504-445f-a078-0a070972b4f2" (UID: "74ab3645-5504-445f-a078-0a070972b4f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:32.739097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.739079    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:32.740959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.740931    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-xks-ca" pod "74ab3645-5504-445f-a078-0a070972b4f2" (UID: "74ab3645-5504-445f-a078-0a070972b4f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:32.740959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.740941    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ab3645-5504-445f-a078-0a070972b4f2-kube-api-access-q4dxc" pod "74ab3645-5504-445f-a078-0a070972b4f2" (UID: "74ab3645-5504-445f-a078-0a070972b4f2"). InnerVolumeSpecName "kube-api-access-q4dxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:32.844217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.841574    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ab3645-5504-445f-a078-0a070972b4f2-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:32.844217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:32.841624    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4dxc\" (UniqueName: \"kubernetes.io/projected/74ab3645-5504-445f-a078-0a070972b4f2-kube-api-access-q4dxc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:33.493386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:33.493355    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"
Oct 30 13:26:33.493536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:33.493384    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw" event={"ID":"74ab3645-5504-445f-a078-0a070972b4f2","Type":"ContainerDied","Data":"ed763f3a61cc349919ffd1ace89163b0ad2630afafc9f7981db7c6ea4c3ae8b0"}
Oct 30 13:26:33.493536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:33.493430    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed763f3a61cc349919ffd1ace89163b0ad2630afafc9f7981db7c6ea4c3ae8b0"
Oct 30 13:26:38.212471 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:26:38.212432    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:26:44.285848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.285812    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"]
Oct 30 13:26:44.286409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.286392    4291 cpu_manager.go:414] "containerMap: removing container" podUID="74ab3645-5504-445f-a078-0a070972b4f2" containerName="revoke-keys"
Oct 30 13:26:44.286409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.286409    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ab3645-5504-445f-a078-0a070972b4f2" containerName="revoke-keys"
Oct 30 13:26:44.286545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.286489    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ab3645-5504-445f-a078-0a070972b4f2" containerName="revoke-keys"
Oct 30 13:26:44.289863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.289844    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.306626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.306598    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"]
Oct 30 13:26:44.314892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.314866    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:26:44.315682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.315664    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:44.315823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.315696    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:26:44.315823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.315716    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:26:44.404157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.404122    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.404344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.404175    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.404409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.404389    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9lwk\" (UniqueName: \"kubernetes.io/projected/790da168-9e8a-4750-bd87-46dccb75570f-kube-api-access-f9lwk\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.505598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.505559    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.505823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.505624    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.505823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.505798    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9lwk\" (UniqueName: \"kubernetes.io/projected/790da168-9e8a-4750-bd87-46dccb75570f-kube-api-access-f9lwk\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.506276 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.506253    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.508020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.507996    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.525004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.524976    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9lwk\" (UniqueName: \"kubernetes.io/projected/790da168-9e8a-4750-bd87-46dccb75570f-kube-api-access-f9lwk\") pod \"maas-api-revoke-sub-ce0a3b3a-27wtw\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.602685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.602620    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:44.615773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.615747    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"]
Oct 30 13:26:44.619956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.619938    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.636550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.636512    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"]
Oct 30 13:26:44.711344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.711293    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.711535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.711360    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n76f\" (UniqueName: \"kubernetes.io/projected/40237e30-dd1f-45e3-96a9-ddeb96b76192-kube-api-access-6n76f\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.711535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.711447    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.812903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.812868    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.813088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.812912    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n76f\" (UniqueName: \"kubernetes.io/projected/40237e30-dd1f-45e3-96a9-ddeb96b76192-kube-api-access-6n76f\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.813088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.812945    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.813811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.813785    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.815779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.815566    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.832159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.832133    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n76f\" (UniqueName: \"kubernetes.io/projected/40237e30-dd1f-45e3-96a9-ddeb96b76192-kube-api-access-6n76f\") pod \"maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:44.922977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.922951    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"]
Oct 30 13:26:44.923846 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:26:44.923818    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod790da168_9e8a_4750_bd87_46dccb75570f.slice/crio-810d123e5f9b616d6de6ae938c442f7eb98c6ab40f083f087de3d65a1fb356d7 WatchSource:0}: Error finding container 810d123e5f9b616d6de6ae938c442f7eb98c6ab40f083f087de3d65a1fb356d7: Status 404 returned error can't find the container with id 810d123e5f9b616d6de6ae938c442f7eb98c6ab40f083f087de3d65a1fb356d7
Oct 30 13:26:44.956516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:44.956492    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:45.268505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:45.268481    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"]
Oct 30 13:26:45.270696 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:26:45.270667    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40237e30_dd1f_45e3_96a9_ddeb96b76192.slice/crio-e88d0044f1c040516694e460458eebccd993f0a9931e8609f44965153eecf68f WatchSource:0}: Error finding container e88d0044f1c040516694e460458eebccd993f0a9931e8609f44965153eecf68f: Status 404 returned error can't find the container with id e88d0044f1c040516694e460458eebccd993f0a9931e8609f44965153eecf68f
Oct 30 13:26:45.587280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:45.587232    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq" event={"ID":"40237e30-dd1f-45e3-96a9-ddeb96b76192","Type":"ContainerStarted","Data":"e88d0044f1c040516694e460458eebccd993f0a9931e8609f44965153eecf68f"}
Oct 30 13:26:45.589324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:45.589292    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw" event={"ID":"790da168-9e8a-4750-bd87-46dccb75570f","Type":"ContainerStarted","Data":"f691a0d033ecf81b887439bb8f5be521d9ea1de2ee049cfda3cdfac54715a1cc"}
Oct 30 13:26:45.589474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:45.589412    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw" event={"ID":"790da168-9e8a-4750-bd87-46dccb75570f","Type":"ContainerStarted","Data":"810d123e5f9b616d6de6ae938c442f7eb98c6ab40f083f087de3d65a1fb356d7"}
Oct 30 13:26:45.628630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:45.628574    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw" podStartSLOduration=1.628560342 podStartE2EDuration="1.628560342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:45.626504911 +0000 UTC m=+1415.886743147" watchObservedRunningTime="2026-10-30 13:26:45.628560342 +0000 UTC m=+1415.888798569"
Oct 30 13:26:45.910219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:45.910183    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:26:46.603002 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:46.602971    4291 generic.go:356] "Generic (PLEG): container finished" podID="40237e30-dd1f-45e3-96a9-ddeb96b76192" containerID="6b12f76f41e126d7acae2d0e418fb3308af0c9ac1bc61c9d900ebb6a65e3d542" exitCode=0
Oct 30 13:26:46.603463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:46.603047    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq" event={"ID":"40237e30-dd1f-45e3-96a9-ddeb96b76192","Type":"ContainerDied","Data":"6b12f76f41e126d7acae2d0e418fb3308af0c9ac1bc61c9d900ebb6a65e3d542"}
Oct 30 13:26:46.605018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:46.604991    4291 generic.go:356] "Generic (PLEG): container finished" podID="790da168-9e8a-4750-bd87-46dccb75570f" containerID="f691a0d033ecf81b887439bb8f5be521d9ea1de2ee049cfda3cdfac54715a1cc" exitCode=0
Oct 30 13:26:46.605145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:46.605071    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw" event={"ID":"790da168-9e8a-4750-bd87-46dccb75570f","Type":"ContainerDied","Data":"f691a0d033ecf81b887439bb8f5be521d9ea1de2ee049cfda3cdfac54715a1cc"}
Oct 30 13:26:48.397497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.389058    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:48.473541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.473515    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:48.541757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.541713    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-xks-ca\") pod \"790da168-9e8a-4750-bd87-46dccb75570f\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") "
Oct 30 13:26:48.541964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.541809    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/790da168-9e8a-4750-bd87-46dccb75570f-kube-api-access-f9lwk\" (UniqueName: \"kubernetes.io/projected/790da168-9e8a-4750-bd87-46dccb75570f-kube-api-access-f9lwk\") pod \"790da168-9e8a-4750-bd87-46dccb75570f\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") "
Oct 30 13:26:48.542042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.542005    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-openshift-ca\") pod \"790da168-9e8a-4750-bd87-46dccb75570f\" (UID: \"790da168-9e8a-4750-bd87-46dccb75570f\") "
Oct 30 13:26:48.543138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.543104    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-openshift-ca" pod "790da168-9e8a-4750-bd87-46dccb75570f" (UID: "790da168-9e8a-4750-bd87-46dccb75570f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:48.544511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.544484    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-xks-ca" pod "790da168-9e8a-4750-bd87-46dccb75570f" (UID: "790da168-9e8a-4750-bd87-46dccb75570f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:48.545103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.545074    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/790da168-9e8a-4750-bd87-46dccb75570f-kube-api-access-f9lwk" pod "790da168-9e8a-4750-bd87-46dccb75570f" (UID: "790da168-9e8a-4750-bd87-46dccb75570f"). InnerVolumeSpecName "kube-api-access-f9lwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:48.635069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.635042    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"
Oct 30 13:26:48.635246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.635072    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw" event={"ID":"790da168-9e8a-4750-bd87-46dccb75570f","Type":"ContainerDied","Data":"810d123e5f9b616d6de6ae938c442f7eb98c6ab40f083f087de3d65a1fb356d7"}
Oct 30 13:26:48.635246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.635116    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="810d123e5f9b616d6de6ae938c442f7eb98c6ab40f083f087de3d65a1fb356d7"
Oct 30 13:26:48.636744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.636727    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"
Oct 30 13:26:48.636884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.636727    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq" event={"ID":"40237e30-dd1f-45e3-96a9-ddeb96b76192","Type":"ContainerDied","Data":"e88d0044f1c040516694e460458eebccd993f0a9931e8609f44965153eecf68f"}
Oct 30 13:26:48.636884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.636837    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e88d0044f1c040516694e460458eebccd993f0a9931e8609f44965153eecf68f"
Oct 30 13:26:48.643342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.643322    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40237e30-dd1f-45e3-96a9-ddeb96b76192-kube-api-access-6n76f\" (UniqueName: \"kubernetes.io/projected/40237e30-dd1f-45e3-96a9-ddeb96b76192-kube-api-access-6n76f\") pod \"40237e30-dd1f-45e3-96a9-ddeb96b76192\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") "
Oct 30 13:26:48.643423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.643404    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-xks-ca\") pod \"40237e30-dd1f-45e3-96a9-ddeb96b76192\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") "
Oct 30 13:26:48.643463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.643433    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-openshift-ca\") pod \"40237e30-dd1f-45e3-96a9-ddeb96b76192\" (UID: \"40237e30-dd1f-45e3-96a9-ddeb96b76192\") "
Oct 30 13:26:48.643844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.643822    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-openshift-ca" pod "40237e30-dd1f-45e3-96a9-ddeb96b76192" (UID: "40237e30-dd1f-45e3-96a9-ddeb96b76192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:48.643914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.643886    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.643914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.643902    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9lwk\" (UniqueName: \"kubernetes.io/projected/790da168-9e8a-4750-bd87-46dccb75570f-kube-api-access-f9lwk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.643973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.643914    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/790da168-9e8a-4750-bd87-46dccb75570f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.645488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.645463    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-xks-ca" pod "40237e30-dd1f-45e3-96a9-ddeb96b76192" (UID: "40237e30-dd1f-45e3-96a9-ddeb96b76192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:48.645601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.645583    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40237e30-dd1f-45e3-96a9-ddeb96b76192-kube-api-access-6n76f" pod "40237e30-dd1f-45e3-96a9-ddeb96b76192" (UID: "40237e30-dd1f-45e3-96a9-ddeb96b76192"). InnerVolumeSpecName "kube-api-access-6n76f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:48.744622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.744591    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n76f\" (UniqueName: \"kubernetes.io/projected/40237e30-dd1f-45e3-96a9-ddeb96b76192-kube-api-access-6n76f\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.744622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.744618    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:48.744622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:48.744627    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40237e30-dd1f-45e3-96a9-ddeb96b76192-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:54.704236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.704201    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"]
Oct 30 13:26:54.704739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.704724    4291 cpu_manager.go:414] "containerMap: removing container" podUID="40237e30-dd1f-45e3-96a9-ddeb96b76192" containerName="revoke-keys"
Oct 30 13:26:54.704816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.704739    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40237e30-dd1f-45e3-96a9-ddeb96b76192" containerName="revoke-keys"
Oct 30 13:26:54.704816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.704762    4291 cpu_manager.go:414] "containerMap: removing container" podUID="790da168-9e8a-4750-bd87-46dccb75570f" containerName="revoke-keys"
Oct 30 13:26:54.704816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.704768    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="790da168-9e8a-4750-bd87-46dccb75570f" containerName="revoke-keys"
Oct 30 13:26:54.704910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.704854    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="40237e30-dd1f-45e3-96a9-ddeb96b76192" containerName="revoke-keys"
Oct 30 13:26:54.704910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.704865    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="790da168-9e8a-4750-bd87-46dccb75570f" containerName="revoke-keys"
Oct 30 13:26:54.708668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.708623    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.721583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.721559    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.721741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.721716    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.721817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.721800    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tljkc\" (UniqueName: \"kubernetes.io/projected/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-kube-api-access-tljkc\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.731203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.731169    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"]
Oct 30 13:26:54.739415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.739242    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:26:54.739415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.739366    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:26:54.802564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.801949    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:26:54.802564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.802089    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:26:54.823418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.823386    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.823607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.823500    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tljkc\" (UniqueName: \"kubernetes.io/projected/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-kube-api-access-tljkc\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.823702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.823647    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.824198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.824177    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.826059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.826036    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:54.843191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:54.843167    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tljkc\" (UniqueName: \"kubernetes.io/projected/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-kube-api-access-tljkc\") pod \"maas-api-revoke-sub-85e3a53a-578lc\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:55.022344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:55.022261    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:55.331464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:55.331427    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"]
Oct 30 13:26:55.331923 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:26:55.331900    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908e07ce_0c83_4d02_8a5a_f10fe785ac5e.slice/crio-1c1f4fd5d55edf32f2cbdd5f2e17831e134ddc350480a4b93b1959c00ed3d474 WatchSource:0}: Error finding container 1c1f4fd5d55edf32f2cbdd5f2e17831e134ddc350480a4b93b1959c00ed3d474: Status 404 returned error can't find the container with id 1c1f4fd5d55edf32f2cbdd5f2e17831e134ddc350480a4b93b1959c00ed3d474
Oct 30 13:26:55.682829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:55.682787    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc" event={"ID":"908e07ce-0c83-4d02-8a5a-f10fe785ac5e","Type":"ContainerStarted","Data":"6ac156cf3411a10fb182e105e7a9fe30f3079e090201214fb62742d24240c8c8"}
Oct 30 13:26:55.682979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:55.682881    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc" event={"ID":"908e07ce-0c83-4d02-8a5a-f10fe785ac5e","Type":"ContainerStarted","Data":"1c1f4fd5d55edf32f2cbdd5f2e17831e134ddc350480a4b93b1959c00ed3d474"}
Oct 30 13:26:55.732152 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:55.732091    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc" podStartSLOduration=1.732078163 podStartE2EDuration="1.732078163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:26:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:26:55.729970981 +0000 UTC m=+1425.990209216" watchObservedRunningTime="2026-10-30 13:26:55.732078163 +0000 UTC m=+1425.992316399"
Oct 30 13:26:56.691472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:56.691424    4291 generic.go:356] "Generic (PLEG): container finished" podID="908e07ce-0c83-4d02-8a5a-f10fe785ac5e" containerID="6ac156cf3411a10fb182e105e7a9fe30f3079e090201214fb62742d24240c8c8" exitCode=0
Oct 30 13:26:56.691658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:56.691486    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc" event={"ID":"908e07ce-0c83-4d02-8a5a-f10fe785ac5e","Type":"ContainerDied","Data":"6ac156cf3411a10fb182e105e7a9fe30f3079e090201214fb62742d24240c8c8"}
Oct 30 13:26:57.868678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.868623    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:57.967226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.967145    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-openshift-ca\") pod \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") "
Oct 30 13:26:57.967226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.967210    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-xks-ca\") pod \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") "
Oct 30 13:26:57.967432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.967325    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-kube-api-access-tljkc\" (UniqueName: \"kubernetes.io/projected/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-kube-api-access-tljkc\") pod \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\" (UID: \"908e07ce-0c83-4d02-8a5a-f10fe785ac5e\") "
Oct 30 13:26:57.967547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.967517    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-openshift-ca" pod "908e07ce-0c83-4d02-8a5a-f10fe785ac5e" (UID: "908e07ce-0c83-4d02-8a5a-f10fe785ac5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:26:57.968038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.968013    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:57.969295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.969270    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-xks-ca" pod "908e07ce-0c83-4d02-8a5a-f10fe785ac5e" (UID: "908e07ce-0c83-4d02-8a5a-f10fe785ac5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:26:57.969414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:57.969374    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-kube-api-access-tljkc" pod "908e07ce-0c83-4d02-8a5a-f10fe785ac5e" (UID: "908e07ce-0c83-4d02-8a5a-f10fe785ac5e"). InnerVolumeSpecName "kube-api-access-tljkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:26:58.069372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:58.069329    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:58.069372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:58.069367    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tljkc\" (UniqueName: \"kubernetes.io/projected/908e07ce-0c83-4d02-8a5a-f10fe785ac5e-kube-api-access-tljkc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:26:58.706931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:58.706899    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"
Oct 30 13:26:58.707086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:58.706938    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc" event={"ID":"908e07ce-0c83-4d02-8a5a-f10fe785ac5e","Type":"ContainerDied","Data":"1c1f4fd5d55edf32f2cbdd5f2e17831e134ddc350480a4b93b1959c00ed3d474"}
Oct 30 13:26:58.707086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:26:58.706994    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1f4fd5d55edf32f2cbdd5f2e17831e134ddc350480a4b93b1959c00ed3d474"
Oct 30 13:27:00.714155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:00.714116    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"]
Oct 30 13:27:00.725587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:00.725553    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-1f486a-mnmxq"]
Oct 30 13:27:01.881025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:01.880987    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40237e30-dd1f-45e3-96a9-ddeb96b76192" path="/var/lib/kubelet/pods/40237e30-dd1f-45e3-96a9-ddeb96b76192/volumes"
Oct 30 13:27:08.211338 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:27:08.211297    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:27:08.271521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.271480    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"]
Oct 30 13:27:08.272206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.272185    4291 cpu_manager.go:414] "containerMap: removing container" podUID="908e07ce-0c83-4d02-8a5a-f10fe785ac5e" containerName="revoke-keys"
Oct 30 13:27:08.272206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.272205    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="908e07ce-0c83-4d02-8a5a-f10fe785ac5e" containerName="revoke-keys"
Oct 30 13:27:08.272339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.272299    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="908e07ce-0c83-4d02-8a5a-f10fe785ac5e" containerName="revoke-keys"
Oct 30 13:27:08.275906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.275887    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.291264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.291226    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:27:08.291537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.291508    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:27:08.296490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.296452    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"]
Oct 30 13:27:08.302768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.302735    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:27:08.302968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.302892    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:27:08.379663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.379598    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lmpg\" (UniqueName: \"kubernetes.io/projected/ffd7e93c-2d21-4666-ac42-ad67543de96c-kube-api-access-5lmpg\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.379895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.379716    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.379982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.379957    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.480924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.480825    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.481103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.480973    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lmpg\" (UniqueName: \"kubernetes.io/projected/ffd7e93c-2d21-4666-ac42-ad67543de96c-kube-api-access-5lmpg\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.481103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.481015    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.481683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.481663    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.483422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.483396    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.502380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.502349    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lmpg\" (UniqueName: \"kubernetes.io/projected/ffd7e93c-2d21-4666-ac42-ad67543de96c-kube-api-access-5lmpg\") pod \"maas-api-revoke-sub-388faeb7-xl5tz\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.590178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.590118    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:08.920060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:08.920032    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"]
Oct 30 13:27:08.922118 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:27:08.922092    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffd7e93c_2d21_4666_ac42_ad67543de96c.slice/crio-a739c1ac63a712829911767524805b4ea566a215cfa35f9a3267026e79b898b3 WatchSource:0}: Error finding container a739c1ac63a712829911767524805b4ea566a215cfa35f9a3267026e79b898b3: Status 404 returned error can't find the container with id a739c1ac63a712829911767524805b4ea566a215cfa35f9a3267026e79b898b3
Oct 30 13:27:09.829052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:09.828992    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz" event={"ID":"ffd7e93c-2d21-4666-ac42-ad67543de96c","Type":"ContainerStarted","Data":"32f16672cbc08533ecbb3866fd8ab61fba5fc576a05a59bb4872eda2eb089987"}
Oct 30 13:27:09.829535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:09.829066    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz" event={"ID":"ffd7e93c-2d21-4666-ac42-ad67543de96c","Type":"ContainerStarted","Data":"a739c1ac63a712829911767524805b4ea566a215cfa35f9a3267026e79b898b3"}
Oct 30 13:27:09.879683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:09.876958    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz" podStartSLOduration=1.876937877 podStartE2EDuration="1.876937877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:09.873364579 +0000 UTC m=+1440.133602814" watchObservedRunningTime="2026-10-30 13:27:09.876937877 +0000 UTC m=+1440.137176115"
Oct 30 13:27:10.837346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:10.837311    4291 generic.go:356] "Generic (PLEG): container finished" podID="ffd7e93c-2d21-4666-ac42-ad67543de96c" containerID="32f16672cbc08533ecbb3866fd8ab61fba5fc576a05a59bb4872eda2eb089987" exitCode=0
Oct 30 13:27:10.837843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:10.837397    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz" event={"ID":"ffd7e93c-2d21-4666-ac42-ad67543de96c","Type":"ContainerDied","Data":"32f16672cbc08533ecbb3866fd8ab61fba5fc576a05a59bb4872eda2eb089987"}
Oct 30 13:27:11.007210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:11.007175    4291 scope.go:122] "RemoveContainer" containerID="427b3944b2e411aa6d3be8f7aeb6801f802822dff7f9fa165dbd7b1665e6d10b"
Oct 30 13:27:11.016890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:11.016867    4291 scope.go:122] "RemoveContainer" containerID="11dfa3ad037128d02a281a15553ebba28c61706d598b275adb7ee5968419df45"
Oct 30 13:27:11.025831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:11.025811    4291 scope.go:122] "RemoveContainer" containerID="0a8b563dc72ae429e854dbc9a9e3a11616bbfaa1af71d7aa7764beb9568747b4"
Oct 30 13:27:12.009347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.009317    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:12.134648 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.134546    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-xks-ca\") pod \"ffd7e93c-2d21-4666-ac42-ad67543de96c\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") "
Oct 30 13:27:12.134814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.134751    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-openshift-ca\") pod \"ffd7e93c-2d21-4666-ac42-ad67543de96c\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") "
Oct 30 13:27:12.134857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.134835    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffd7e93c-2d21-4666-ac42-ad67543de96c-kube-api-access-5lmpg\" (UniqueName: \"kubernetes.io/projected/ffd7e93c-2d21-4666-ac42-ad67543de96c-kube-api-access-5lmpg\") pod \"ffd7e93c-2d21-4666-ac42-ad67543de96c\" (UID: \"ffd7e93c-2d21-4666-ac42-ad67543de96c\") "
Oct 30 13:27:12.135096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.135069    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-openshift-ca" pod "ffd7e93c-2d21-4666-ac42-ad67543de96c" (UID: "ffd7e93c-2d21-4666-ac42-ad67543de96c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:12.135496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.135476    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:12.137014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.136982    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-xks-ca" pod "ffd7e93c-2d21-4666-ac42-ad67543de96c" (UID: "ffd7e93c-2d21-4666-ac42-ad67543de96c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:12.137133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.137055    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffd7e93c-2d21-4666-ac42-ad67543de96c-kube-api-access-5lmpg" pod "ffd7e93c-2d21-4666-ac42-ad67543de96c" (UID: "ffd7e93c-2d21-4666-ac42-ad67543de96c"). InnerVolumeSpecName "kube-api-access-5lmpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:12.237272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.237236    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lmpg\" (UniqueName: \"kubernetes.io/projected/ffd7e93c-2d21-4666-ac42-ad67543de96c-kube-api-access-5lmpg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:12.237272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.237268    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ffd7e93c-2d21-4666-ac42-ad67543de96c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:12.854002 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.853970    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"
Oct 30 13:27:12.854219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.854000    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz" event={"ID":"ffd7e93c-2d21-4666-ac42-ad67543de96c","Type":"ContainerDied","Data":"a739c1ac63a712829911767524805b4ea566a215cfa35f9a3267026e79b898b3"}
Oct 30 13:27:12.854219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:12.854048    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a739c1ac63a712829911767524805b4ea566a215cfa35f9a3267026e79b898b3"
Oct 30 13:27:15.290298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.290260    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"]
Oct 30 13:27:15.290852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.290835    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ffd7e93c-2d21-4666-ac42-ad67543de96c" containerName="revoke-keys"
Oct 30 13:27:15.290901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.290853    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffd7e93c-2d21-4666-ac42-ad67543de96c" containerName="revoke-keys"
Oct 30 13:27:15.290971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.290962    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffd7e93c-2d21-4666-ac42-ad67543de96c" containerName="revoke-keys"
Oct 30 13:27:15.294447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.294432    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.310392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.310359    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"]
Oct 30 13:27:15.318618 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.318589    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:27:15.318863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.318692    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:27:15.318863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.318715    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:27:15.320667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.320617    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:27:15.384697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.384598    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.384921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.384801    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnh76\" (UniqueName: \"kubernetes.io/projected/ae2751f1-0212-4ff2-a299-51f84a9013b5-kube-api-access-pnh76\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.384985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.384937    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.486710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.486670    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.486883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.486806    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnh76\" (UniqueName: \"kubernetes.io/projected/ae2751f1-0212-4ff2-a299-51f84a9013b5-kube-api-access-pnh76\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.486923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.486898    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.487320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.487302    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.489157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.489136    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.506622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.506598    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnh76\" (UniqueName: \"kubernetes.io/projected/ae2751f1-0212-4ff2-a299-51f84a9013b5-kube-api-access-pnh76\") pod \"maas-api-revoke-sub-20ced213-x7mrt\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.608708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.608663    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:15.998029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:15.995017    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"]
Oct 30 13:27:16.997000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:16.996966    4291 generic.go:356] "Generic (PLEG): container finished" podID="ae2751f1-0212-4ff2-a299-51f84a9013b5" containerID="cfe014a874c2612b84f9ddd7d576d01b656d4e79392d37d352bcd97360487b49" exitCode=0
Oct 30 13:27:16.997424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:16.997050    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt" event={"ID":"ae2751f1-0212-4ff2-a299-51f84a9013b5","Type":"ContainerDied","Data":"cfe014a874c2612b84f9ddd7d576d01b656d4e79392d37d352bcd97360487b49"}
Oct 30 13:27:16.997424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:16.997115    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt" event={"ID":"ae2751f1-0212-4ff2-a299-51f84a9013b5","Type":"ContainerStarted","Data":"3b0dcc03c9919fd830638744f85f459519e401dea9db83b6861e3f5929b96569"}
Oct 30 13:27:18.159194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.159164    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:18.244454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.244418    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-xks-ca\") pod \"ae2751f1-0212-4ff2-a299-51f84a9013b5\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") "
Oct 30 13:27:18.244672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.244579    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-openshift-ca\") pod \"ae2751f1-0212-4ff2-a299-51f84a9013b5\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") "
Oct 30 13:27:18.244672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.244611    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae2751f1-0212-4ff2-a299-51f84a9013b5-kube-api-access-pnh76\" (UniqueName: \"kubernetes.io/projected/ae2751f1-0212-4ff2-a299-51f84a9013b5-kube-api-access-pnh76\") pod \"ae2751f1-0212-4ff2-a299-51f84a9013b5\" (UID: \"ae2751f1-0212-4ff2-a299-51f84a9013b5\") "
Oct 30 13:27:18.245032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.245000    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-openshift-ca" pod "ae2751f1-0212-4ff2-a299-51f84a9013b5" (UID: "ae2751f1-0212-4ff2-a299-51f84a9013b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:18.245257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.245238    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.246807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.246782    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-xks-ca" pod "ae2751f1-0212-4ff2-a299-51f84a9013b5" (UID: "ae2751f1-0212-4ff2-a299-51f84a9013b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:18.246880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.246838    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae2751f1-0212-4ff2-a299-51f84a9013b5-kube-api-access-pnh76" pod "ae2751f1-0212-4ff2-a299-51f84a9013b5" (UID: "ae2751f1-0212-4ff2-a299-51f84a9013b5"). InnerVolumeSpecName "kube-api-access-pnh76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:18.346838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.346791    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae2751f1-0212-4ff2-a299-51f84a9013b5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.346838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.346819    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnh76\" (UniqueName: \"kubernetes.io/projected/ae2751f1-0212-4ff2-a299-51f84a9013b5-kube-api-access-pnh76\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:18.640876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.640840    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"]
Oct 30 13:27:18.641380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.641365    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ae2751f1-0212-4ff2-a299-51f84a9013b5" containerName="revoke-keys"
Oct 30 13:27:18.641426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.641381    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae2751f1-0212-4ff2-a299-51f84a9013b5" containerName="revoke-keys"
Oct 30 13:27:18.641501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.641492    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae2751f1-0212-4ff2-a299-51f84a9013b5" containerName="revoke-keys"
Oct 30 13:27:18.644941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.644925    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.662501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.662470    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"]
Oct 30 13:27:18.751175 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.751129    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.751363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.751245    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.751437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.751375    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnb2\" (UniqueName: \"kubernetes.io/projected/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-kube-api-access-jlnb2\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.852853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.852817    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnb2\" (UniqueName: \"kubernetes.io/projected/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-kube-api-access-jlnb2\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.853038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.852960    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.853081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.853049    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.853731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.853713    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.855469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.855449    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.874744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.874698    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnb2\" (UniqueName: \"kubernetes.io/projected/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-kube-api-access-jlnb2\") pod \"maas-api-revoke-sub-d6fc2635-rsl7l\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:18.959473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:18.959393    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:19.013971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:19.013914    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt" event={"ID":"ae2751f1-0212-4ff2-a299-51f84a9013b5","Type":"ContainerDied","Data":"3b0dcc03c9919fd830638744f85f459519e401dea9db83b6861e3f5929b96569"}
Oct 30 13:27:19.014171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:19.014023    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3b0dcc03c9919fd830638744f85f459519e401dea9db83b6861e3f5929b96569"
Oct 30 13:27:19.014171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:19.014102    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"
Oct 30 13:27:19.281808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:19.281776    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"]
Oct 30 13:27:19.283757 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:27:19.283730    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1cce30_bbe1_4c9d_aae5_fd717fcd7ca7.slice/crio-dad746a5a031ee565553634ac777656450e978bc9fa5773b2a5137444d1dd6a4 WatchSource:0}: Error finding container dad746a5a031ee565553634ac777656450e978bc9fa5773b2a5137444d1dd6a4: Status 404 returned error can't find the container with id dad746a5a031ee565553634ac777656450e978bc9fa5773b2a5137444d1dd6a4
Oct 30 13:27:20.023020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:20.022976    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l" event={"ID":"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7","Type":"ContainerStarted","Data":"9d34cbe0286e3974c653d4dc012fb355e08c43af5881ffddf9a2750b0756f862"}
Oct 30 13:27:20.023190 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:20.023044    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l" event={"ID":"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7","Type":"ContainerStarted","Data":"dad746a5a031ee565553634ac777656450e978bc9fa5773b2a5137444d1dd6a4"}
Oct 30 13:27:20.062940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:20.062887    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l" podStartSLOduration=2.062870703 podStartE2EDuration="2.062870703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:20.062097369 +0000 UTC m=+1450.322335604" watchObservedRunningTime="2026-10-30 13:27:20.062870703 +0000 UTC m=+1450.323108937"
Oct 30 13:27:21.031518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:21.031479    4291 generic.go:356] "Generic (PLEG): container finished" podID="dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" containerID="9d34cbe0286e3974c653d4dc012fb355e08c43af5881ffddf9a2750b0756f862" exitCode=0
Oct 30 13:27:21.031947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:21.031566    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l" event={"ID":"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7","Type":"ContainerDied","Data":"9d34cbe0286e3974c653d4dc012fb355e08c43af5881ffddf9a2750b0756f862"}
Oct 30 13:27:22.202331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.202304    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:22.297061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.296952    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-xks-ca\") pod \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") "
Oct 30 13:27:22.297255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.297094    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-openshift-ca\") pod \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") "
Oct 30 13:27:22.297331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.297261    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-kube-api-access-jlnb2\" (UniqueName: \"kubernetes.io/projected/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-kube-api-access-jlnb2\") pod \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\" (UID: \"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7\") "
Oct 30 13:27:22.297532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.297500    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-openshift-ca" pod "dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" (UID: "dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:22.297971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.297945    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:22.299166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.299133    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-xks-ca" pod "dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" (UID: "dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:22.299308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.299291    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-kube-api-access-jlnb2" pod "dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" (UID: "dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7"). InnerVolumeSpecName "kube-api-access-jlnb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:22.400204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.400163    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlnb2\" (UniqueName: \"kubernetes.io/projected/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-kube-api-access-jlnb2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:22.400204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.400201    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:22.876268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.876237    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"]
Oct 30 13:27:22.876830 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.876813    4291 cpu_manager.go:414] "containerMap: removing container" podUID="dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" containerName="revoke-keys"
Oct 30 13:27:22.876891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.876832    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" containerName="revoke-keys"
Oct 30 13:27:22.876943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.876934    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" containerName="revoke-keys"
Oct 30 13:27:22.880504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.880486    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:22.897852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:22.897825    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"]
Oct 30 13:27:23.008473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.008430    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.008715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.008604    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.008804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.008728    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqmtz\" (UniqueName: \"kubernetes.io/projected/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-kube-api-access-tqmtz\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.046167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.046134    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"
Oct 30 13:27:23.046358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.046165    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l" event={"ID":"dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7","Type":"ContainerDied","Data":"dad746a5a031ee565553634ac777656450e978bc9fa5773b2a5137444d1dd6a4"}
Oct 30 13:27:23.046358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.046214    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad746a5a031ee565553634ac777656450e978bc9fa5773b2a5137444d1dd6a4"
Oct 30 13:27:23.110192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.110157    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.110358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.110231    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqmtz\" (UniqueName: \"kubernetes.io/projected/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-kube-api-access-tqmtz\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.110358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.110340    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.110856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.110832    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.112725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.112705    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.130621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.130562    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqmtz\" (UniqueName: \"kubernetes.io/projected/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-kube-api-access-tqmtz\") pod \"maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.200874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.200836    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:23.509664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:23.509626    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"]
Oct 30 13:27:23.511780 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:27:23.511750    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a1bd9b8_f897_4987_b9fc_4cc4f757ae8a.slice/crio-4db323137a9bb00ad0e9260addb66cf8a860c6053f9a34635aa317f7f9a2efe3 WatchSource:0}: Error finding container 4db323137a9bb00ad0e9260addb66cf8a860c6053f9a34635aa317f7f9a2efe3: Status 404 returned error can't find the container with id 4db323137a9bb00ad0e9260addb66cf8a860c6053f9a34635aa317f7f9a2efe3
Oct 30 13:27:24.055103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:24.055008    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9" event={"ID":"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a","Type":"ContainerStarted","Data":"6f0615fc60150431eaf88932384e21ed8fba3f62d78f9f4928b86ed6763551a8"}
Oct 30 13:27:24.055103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:24.055072    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9" event={"ID":"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a","Type":"ContainerStarted","Data":"4db323137a9bb00ad0e9260addb66cf8a860c6053f9a34635aa317f7f9a2efe3"}
Oct 30 13:27:24.102454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:24.102394    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9" podStartSLOduration=2.102380184 podStartE2EDuration="2.102380184s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:24.100199213 +0000 UTC m=+1454.360437447" watchObservedRunningTime="2026-10-30 13:27:24.102380184 +0000 UTC m=+1454.362618417"
Oct 30 13:27:25.063819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:25.063786    4291 generic.go:356] "Generic (PLEG): container finished" podID="3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" containerID="6f0615fc60150431eaf88932384e21ed8fba3f62d78f9f4928b86ed6763551a8" exitCode=0
Oct 30 13:27:25.064225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:25.063881    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9" event={"ID":"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a","Type":"ContainerDied","Data":"6f0615fc60150431eaf88932384e21ed8fba3f62d78f9f4928b86ed6763551a8"}
Oct 30 13:27:26.236838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.236810    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:26.353011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.352977    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-xks-ca\") pod \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") "
Oct 30 13:27:26.353202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.353028    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-openshift-ca\") pod \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") "
Oct 30 13:27:26.353202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.353169    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-kube-api-access-tqmtz\" (UniqueName: \"kubernetes.io/projected/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-kube-api-access-tqmtz\") pod \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\" (UID: \"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a\") "
Oct 30 13:27:26.353438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.353410    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-openshift-ca" pod "3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" (UID: "3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:26.353702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.353678    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:26.355120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.355099    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-xks-ca" pod "3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" (UID: "3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:26.355242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.355227    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-kube-api-access-tqmtz" pod "3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" (UID: "3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a"). InnerVolumeSpecName "kube-api-access-tqmtz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:26.455079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.455040    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:26.455079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:26.455072    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqmtz\" (UniqueName: \"kubernetes.io/projected/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a-kube-api-access-tqmtz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:27.083008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:27.082965    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9" event={"ID":"3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a","Type":"ContainerDied","Data":"4db323137a9bb00ad0e9260addb66cf8a860c6053f9a34635aa317f7f9a2efe3"}
Oct 30 13:27:27.083281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:27.083036    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4db323137a9bb00ad0e9260addb66cf8a860c6053f9a34635aa317f7f9a2efe3"
Oct 30 13:27:27.083281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:27.082980    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"
Oct 30 13:27:27.641164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:27.641126    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:27:30.781660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:30.779018    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:27:30.875782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:30.875733    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" probeResult="failure" output="Get \"https://10.133.0.55:8000/ready\": context deadline exceeded"
Oct 30 13:27:30.882780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:30.882723    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:27:31.138202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:31.138161    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded"
Oct 30 13:27:31.211777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:31.211726    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" probeResult="failure" output="Get \"https://10.133.0.41:8000/ready\": context deadline exceeded"
Oct 30 13:27:31.977308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:31.977271    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"]
Oct 30 13:27:31.977854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:31.977837    4291 cpu_manager.go:414] "containerMap: removing container" podUID="3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" containerName="revoke-keys"
Oct 30 13:27:31.977901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:31.977855    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" containerName="revoke-keys"
Oct 30 13:27:31.977967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:31.977958    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" containerName="revoke-keys"
Oct 30 13:27:31.981341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:31.981323    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.012651 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.012610    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:27:32.012811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.012699    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:27:32.012811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.012712    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:27:32.012811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.012710    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:27:32.018856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.018828    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"]
Oct 30 13:27:32.141610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.141574    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.141919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.141891    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.142111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.142093    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stz52\" (UniqueName: \"kubernetes.io/projected/3c36da58-217f-4527-ae20-86b987f254c6-kube-api-access-stz52\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.243387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.243295    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.243387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.243339    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.243597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.243392    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stz52\" (UniqueName: \"kubernetes.io/projected/3c36da58-217f-4527-ae20-86b987f254c6-kube-api-access-stz52\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.243979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.243952    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.245922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.245897    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.270312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.270281    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stz52\" (UniqueName: \"kubernetes.io/projected/3c36da58-217f-4527-ae20-86b987f254c6-kube-api-access-stz52\") pod \"maas-api-revoke-sub-d4b54114-xrvlg\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.295244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.295195    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:32.764058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.764028    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"]
Oct 30 13:27:32.768228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.768201    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.792978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.792948    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"]
Oct 30 13:27:32.813123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.813087    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"]
Oct 30 13:27:32.818084 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:27:32.818045    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c36da58_217f_4527_ae20_86b987f254c6.slice/crio-53d3c3186fba77633bb3487079af2ade4ed485e7fbe88aa38fed662b395b76f6 WatchSource:0}: Error finding container 53d3c3186fba77633bb3487079af2ade4ed485e7fbe88aa38fed662b395b76f6: Status 404 returned error can't find the container with id 53d3c3186fba77633bb3487079af2ade4ed485e7fbe88aa38fed662b395b76f6
Oct 30 13:27:32.829744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.829716    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kmk2\" (UniqueName: \"kubernetes.io/projected/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-kube-api-access-9kmk2\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.829887 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.829868    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.829960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.829934    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.931562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.931525    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kmk2\" (UniqueName: \"kubernetes.io/projected/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-kube-api-access-9kmk2\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.931825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.931701    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.931825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.931766    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.932477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.932455    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.934096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.934067    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:32.957815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:32.957790    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kmk2\" (UniqueName: \"kubernetes.io/projected/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-kube-api-access-9kmk2\") pod \"maas-api-revoke-sub-0d487447-wgc4r\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:33.083421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.083381    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:33.188574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.187613    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg" event={"ID":"3c36da58-217f-4527-ae20-86b987f254c6","Type":"ContainerStarted","Data":"2f7175c36f64d03f3435f365c48984cc7dee006b12b3dd22254eb9549a65d5cf"}
Oct 30 13:27:33.188574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.187693    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg" event={"ID":"3c36da58-217f-4527-ae20-86b987f254c6","Type":"ContainerStarted","Data":"53d3c3186fba77633bb3487079af2ade4ed485e7fbe88aa38fed662b395b76f6"}
Oct 30 13:27:33.210687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.210650    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"]
Oct 30 13:27:33.214918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.214895    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.236361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.236326    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltvsj\" (UniqueName: \"kubernetes.io/projected/94f183c2-fa1a-4620-9812-0898de7e146c-kube-api-access-ltvsj\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.236522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.236454    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.236581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.236565    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.237606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.237586    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"]
Oct 30 13:27:33.245566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.245498    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg" podStartSLOduration=2.245478623 podStartE2EDuration="2.245478623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:33.241435042 +0000 UTC m=+1463.501673284" watchObservedRunningTime="2026-10-30 13:27:33.245478623 +0000 UTC m=+1463.505716858"
Oct 30 13:27:33.338301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.338209    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltvsj\" (UniqueName: \"kubernetes.io/projected/94f183c2-fa1a-4620-9812-0898de7e146c-kube-api-access-ltvsj\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.338301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.338282    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.338530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.338353    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.339106 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.339085    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.340917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.340889    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.367060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.367032    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltvsj\" (UniqueName: \"kubernetes.io/projected/94f183c2-fa1a-4620-9812-0898de7e146c-kube-api-access-ltvsj\") pod \"maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.392339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.392311    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"]
Oct 30 13:27:33.393206 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:27:33.393167    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07c9487_95d2_4da5_b8f2_a3f0e46f33f1.slice/crio-4b2dcdd9d893bf990301f851e5a6d972dcf54868e8f0829711551632d045d421 WatchSource:0}: Error finding container 4b2dcdd9d893bf990301f851e5a6d972dcf54868e8f0829711551632d045d421: Status 404 returned error can't find the container with id 4b2dcdd9d893bf990301f851e5a6d972dcf54868e8f0829711551632d045d421
Oct 30 13:27:33.529175 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.529135    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:33.731838 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:27:33.731796    4291 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c36da58_217f_4527_ae20_86b987f254c6.slice/crio-conmon-2f7175c36f64d03f3435f365c48984cc7dee006b12b3dd22254eb9549a65d5cf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c36da58_217f_4527_ae20_86b987f254c6.slice/crio-2f7175c36f64d03f3435f365c48984cc7dee006b12b3dd22254eb9549a65d5cf.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:27:33.861928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:33.861874    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"]
Oct 30 13:27:33.863341 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:27:33.863306    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94f183c2_fa1a_4620_9812_0898de7e146c.slice/crio-c2c54c4c23e8c8451a09e7084946d04295d1d14493c79d5c1d3c2db4722c6cfb WatchSource:0}: Error finding container c2c54c4c23e8c8451a09e7084946d04295d1d14493c79d5c1d3c2db4722c6cfb: Status 404 returned error can't find the container with id c2c54c4c23e8c8451a09e7084946d04295d1d14493c79d5c1d3c2db4722c6cfb
Oct 30 13:27:34.196051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.196014    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r" event={"ID":"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1","Type":"ContainerStarted","Data":"8edd27205c1edbf08ca8ff4f75ba8fb5f8676c1d59149039dc0515df53efb652"}
Oct 30 13:27:34.196564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.196083    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r" event={"ID":"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1","Type":"ContainerStarted","Data":"4b2dcdd9d893bf990301f851e5a6d972dcf54868e8f0829711551632d045d421"}
Oct 30 13:27:34.197932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.197906    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx" event={"ID":"94f183c2-fa1a-4620-9812-0898de7e146c","Type":"ContainerStarted","Data":"a474256b66012db897ca927531b95a100caba5ba9c9cb8bb463c8e37a9f284be"}
Oct 30 13:27:34.198065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.197949    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx" event={"ID":"94f183c2-fa1a-4620-9812-0898de7e146c","Type":"ContainerStarted","Data":"c2c54c4c23e8c8451a09e7084946d04295d1d14493c79d5c1d3c2db4722c6cfb"}
Oct 30 13:27:34.202365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.200488    4291 generic.go:356] "Generic (PLEG): container finished" podID="3c36da58-217f-4527-ae20-86b987f254c6" containerID="2f7175c36f64d03f3435f365c48984cc7dee006b12b3dd22254eb9549a65d5cf" exitCode=0
Oct 30 13:27:34.202365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.200564    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg" event={"ID":"3c36da58-217f-4527-ae20-86b987f254c6","Type":"ContainerDied","Data":"2f7175c36f64d03f3435f365c48984cc7dee006b12b3dd22254eb9549a65d5cf"}
Oct 30 13:27:34.253705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.247845    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r" podStartSLOduration=2.247822729 podStartE2EDuration="2.247822729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:34.243953687 +0000 UTC m=+1464.504191923" watchObservedRunningTime="2026-10-30 13:27:34.247822729 +0000 UTC m=+1464.508060964"
Oct 30 13:27:34.289394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:34.289340    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx" podStartSLOduration=1.289324611 podStartE2EDuration="1.289324611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:34.287525481 +0000 UTC m=+1464.547763733" watchObservedRunningTime="2026-10-30 13:27:34.289324611 +0000 UTC m=+1464.549562910"
Oct 30 13:27:35.208834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.208800    4291 generic.go:356] "Generic (PLEG): container finished" podID="94f183c2-fa1a-4620-9812-0898de7e146c" containerID="a474256b66012db897ca927531b95a100caba5ba9c9cb8bb463c8e37a9f284be" exitCode=0
Oct 30 13:27:35.209278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.208882    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx" event={"ID":"94f183c2-fa1a-4620-9812-0898de7e146c","Type":"ContainerDied","Data":"a474256b66012db897ca927531b95a100caba5ba9c9cb8bb463c8e37a9f284be"}
Oct 30 13:27:35.210577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.210549    4291 generic.go:356] "Generic (PLEG): container finished" podID="d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" containerID="8edd27205c1edbf08ca8ff4f75ba8fb5f8676c1d59149039dc0515df53efb652" exitCode=0
Oct 30 13:27:35.210721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.210615    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r" event={"ID":"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1","Type":"ContainerDied","Data":"8edd27205c1edbf08ca8ff4f75ba8fb5f8676c1d59149039dc0515df53efb652"}
Oct 30 13:27:35.378116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.378089    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:35.471586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.471505    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c36da58-217f-4527-ae20-86b987f254c6-kube-api-access-stz52\" (UniqueName: \"kubernetes.io/projected/3c36da58-217f-4527-ae20-86b987f254c6-kube-api-access-stz52\") pod \"3c36da58-217f-4527-ae20-86b987f254c6\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") "
Oct 30 13:27:35.471772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.471677    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-xks-ca\") pod \"3c36da58-217f-4527-ae20-86b987f254c6\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") "
Oct 30 13:27:35.471835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.471821    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-openshift-ca\") pod \"3c36da58-217f-4527-ae20-86b987f254c6\" (UID: \"3c36da58-217f-4527-ae20-86b987f254c6\") "
Oct 30 13:27:35.472187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.472160    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-openshift-ca" pod "3c36da58-217f-4527-ae20-86b987f254c6" (UID: "3c36da58-217f-4527-ae20-86b987f254c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:35.472420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.472404    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:35.473553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.473527    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c36da58-217f-4527-ae20-86b987f254c6-kube-api-access-stz52" pod "3c36da58-217f-4527-ae20-86b987f254c6" (UID: "3c36da58-217f-4527-ae20-86b987f254c6"). InnerVolumeSpecName "kube-api-access-stz52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:35.473768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.473753    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-xks-ca" pod "3c36da58-217f-4527-ae20-86b987f254c6" (UID: "3c36da58-217f-4527-ae20-86b987f254c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:35.573602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.573563    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c36da58-217f-4527-ae20-86b987f254c6-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:35.573602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:35.573599    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stz52\" (UniqueName: \"kubernetes.io/projected/3c36da58-217f-4527-ae20-86b987f254c6-kube-api-access-stz52\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:36.218362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.218270    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"
Oct 30 13:27:36.218362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.218298    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg" event={"ID":"3c36da58-217f-4527-ae20-86b987f254c6","Type":"ContainerDied","Data":"53d3c3186fba77633bb3487079af2ade4ed485e7fbe88aa38fed662b395b76f6"}
Oct 30 13:27:36.218362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.218352    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53d3c3186fba77633bb3487079af2ade4ed485e7fbe88aa38fed662b395b76f6"
Oct 30 13:27:36.385925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.385576    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:36.421558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.421530    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:36.487096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487009    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-xks-ca\") pod \"94f183c2-fa1a-4620-9812-0898de7e146c\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") "
Oct 30 13:27:36.487096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487045    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-openshift-ca\") pod \"94f183c2-fa1a-4620-9812-0898de7e146c\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") "
Oct 30 13:27:36.487096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487090    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94f183c2-fa1a-4620-9812-0898de7e146c-kube-api-access-ltvsj\" (UniqueName: \"kubernetes.io/projected/94f183c2-fa1a-4620-9812-0898de7e146c-kube-api-access-ltvsj\") pod \"94f183c2-fa1a-4620-9812-0898de7e146c\" (UID: \"94f183c2-fa1a-4620-9812-0898de7e146c\") "
Oct 30 13:27:36.487388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487290    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-xks-ca\") pod \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") "
Oct 30 13:27:36.487494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487454    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-kube-api-access-9kmk2\" (UniqueName: \"kubernetes.io/projected/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-kube-api-access-9kmk2\") pod \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") "
Oct 30 13:27:36.487568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487509    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-openshift-ca\") pod \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\" (UID: \"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1\") "
Oct 30 13:27:36.487568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487524    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-openshift-ca" pod "94f183c2-fa1a-4620-9812-0898de7e146c" (UID: "94f183c2-fa1a-4620-9812-0898de7e146c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:36.487982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.487956    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-openshift-ca" pod "d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" (UID: "d07c9487-95d2-4da5-b8f2-a3f0e46f33f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:27:36.488133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.488119    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:36.488196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.488136    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:36.490033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.490005    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-kube-api-access-9kmk2" pod "d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" (UID: "d07c9487-95d2-4da5-b8f2-a3f0e46f33f1"). InnerVolumeSpecName "kube-api-access-9kmk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:36.490162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.490140    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-xks-ca" pod "94f183c2-fa1a-4620-9812-0898de7e146c" (UID: "94f183c2-fa1a-4620-9812-0898de7e146c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:36.490390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.490355    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-xks-ca" pod "d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" (UID: "d07c9487-95d2-4da5-b8f2-a3f0e46f33f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:27:36.491374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.491355    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94f183c2-fa1a-4620-9812-0898de7e146c-kube-api-access-ltvsj" pod "94f183c2-fa1a-4620-9812-0898de7e146c" (UID: "94f183c2-fa1a-4620-9812-0898de7e146c"). InnerVolumeSpecName "kube-api-access-ltvsj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:27:36.589558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.589518    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kmk2\" (UniqueName: \"kubernetes.io/projected/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-kube-api-access-9kmk2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:36.589558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.589551    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94f183c2-fa1a-4620-9812-0898de7e146c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:36.589558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.589564    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltvsj\" (UniqueName: \"kubernetes.io/projected/94f183c2-fa1a-4620-9812-0898de7e146c-kube-api-access-ltvsj\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:36.589832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:36.589576    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:27:37.227058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:37.227016    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx" event={"ID":"94f183c2-fa1a-4620-9812-0898de7e146c","Type":"ContainerDied","Data":"c2c54c4c23e8c8451a09e7084946d04295d1d14493c79d5c1d3c2db4722c6cfb"}
Oct 30 13:27:37.227493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:37.227082    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c2c54c4c23e8c8451a09e7084946d04295d1d14493c79d5c1d3c2db4722c6cfb"
Oct 30 13:27:37.227493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:37.227031    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"
Oct 30 13:27:37.228580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:37.228546    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r" event={"ID":"d07c9487-95d2-4da5-b8f2-a3f0e46f33f1","Type":"ContainerDied","Data":"4b2dcdd9d893bf990301f851e5a6d972dcf54868e8f0829711551632d045d421"}
Oct 30 13:27:37.228723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:37.228589    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"
Oct 30 13:27:37.228723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:37.228601    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2dcdd9d893bf990301f851e5a6d972dcf54868e8f0829711551632d045d421"
Oct 30 13:27:38.210757 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:27:38.210715    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:27:46.705647 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:46.705606    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"]
Oct 30 13:27:46.718099 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:46.718067    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-d797fc-c6vz9"]
Oct 30 13:27:47.879028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:47.878996    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a" path="/var/lib/kubelet/pods/3a1bd9b8-f897-4987-b9fc-4cc4f757ae8a/volumes"
Oct 30 13:27:53.755714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:53.752827    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" probeResult="failure" output="Get \"https://10.133.0.57:8000/ready\": context deadline exceeded"
Oct 30 13:27:57.276436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.276399    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"]
Oct 30 13:27:57.276979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.276959    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" containerName="revoke-keys"
Oct 30 13:27:57.276979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.276979    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" containerName="revoke-keys"
Oct 30 13:27:57.277062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.277015    4291 cpu_manager.go:414] "containerMap: removing container" podUID="3c36da58-217f-4527-ae20-86b987f254c6" containerName="revoke-keys"
Oct 30 13:27:57.277062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.277022    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c36da58-217f-4527-ae20-86b987f254c6" containerName="revoke-keys"
Oct 30 13:27:57.277062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.277038    4291 cpu_manager.go:414] "containerMap: removing container" podUID="94f183c2-fa1a-4620-9812-0898de7e146c" containerName="revoke-keys"
Oct 30 13:27:57.277062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.277044    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94f183c2-fa1a-4620-9812-0898de7e146c" containerName="revoke-keys"
Oct 30 13:27:57.277213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.277130    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c36da58-217f-4527-ae20-86b987f254c6" containerName="revoke-keys"
Oct 30 13:27:57.277213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.277140    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="94f183c2-fa1a-4620-9812-0898de7e146c" containerName="revoke-keys"
Oct 30 13:27:57.277213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.277147    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" containerName="revoke-keys"
Oct 30 13:27:57.282058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.282036    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.304438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.304400    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"]
Oct 30 13:27:57.310615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.310571    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:27:57.310802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.310688    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:27:57.310802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.310689    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:27:57.310913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.310824    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:27:57.428869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.428827    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.428869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.428874    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbpc5\" (UniqueName: \"kubernetes.io/projected/1f033bf8-00e6-4882-90fc-011782035d5e-kube-api-access-fbpc5\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.429108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.429016    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.530668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.530572    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.530834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.530819    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.530877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.530843    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbpc5\" (UniqueName: \"kubernetes.io/projected/1f033bf8-00e6-4882-90fc-011782035d5e-kube-api-access-fbpc5\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.531376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.531355    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.533079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.533059    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.558883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.558846    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbpc5\" (UniqueName: \"kubernetes.io/projected/1f033bf8-00e6-4882-90fc-011782035d5e-kube-api-access-fbpc5\") pod \"maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.600682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.600647    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:27:57.830357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.830266    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"]
Oct 30 13:27:57.835480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.835458    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:57.897539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.897488    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"]
Oct 30 13:27:57.937592 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.937555    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:57.937823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.937799    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:57.937891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.937840    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-982gt\" (UniqueName: \"kubernetes.io/projected/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-kube-api-access-982gt\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:57.946595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:57.946565    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"]
Oct 30 13:27:58.039553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.039520    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:58.039766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.039663    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:58.039766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.039688    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-982gt\" (UniqueName: \"kubernetes.io/projected/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-kube-api-access-982gt\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:58.040225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.040202    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:58.041979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.041954    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:58.068840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.068805    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-982gt\" (UniqueName: \"kubernetes.io/projected/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-kube-api-access-982gt\") pod \"maas-api-revoke-sub-9f9b336c-hd8wl\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:58.150884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.150849    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:27:58.530558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.530528    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"]
Oct 30 13:27:58.531647 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:27:58.531594    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf50c5c52_6fb8_4c5c_81b5_4242707a1da8.slice/crio-04b732e07dce2f6f84e6c39a750950242162bcdad9db1d79dc0e20a5896c3f76 WatchSource:0}: Error finding container 04b732e07dce2f6f84e6c39a750950242162bcdad9db1d79dc0e20a5896c3f76: Status 404 returned error can't find the container with id 04b732e07dce2f6f84e6c39a750950242162bcdad9db1d79dc0e20a5896c3f76
Oct 30 13:27:58.585432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.585376    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl" event={"ID":"f50c5c52-6fb8-4c5c-81b5-4242707a1da8","Type":"ContainerStarted","Data":"04b732e07dce2f6f84e6c39a750950242162bcdad9db1d79dc0e20a5896c3f76"}
Oct 30 13:27:58.587186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.587154    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt" event={"ID":"1f033bf8-00e6-4882-90fc-011782035d5e","Type":"ContainerStarted","Data":"14c5dfa24b3594516241b0cc2bb3ff68e30be925466d549a114a299b2f9c3255"}
Oct 30 13:27:58.587317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.587207    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt" event={"ID":"1f033bf8-00e6-4882-90fc-011782035d5e","Type":"ContainerStarted","Data":"fd0eecdab45518a6c2a5456f71916c125f13e7f7ca3222e537ffdfc0cec8c929"}
Oct 30 13:27:58.631449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:58.631396    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt" podStartSLOduration=1.6313809639999999 podStartE2EDuration="1.631380964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:27:58.628418809 +0000 UTC m=+1488.888657043" watchObservedRunningTime="2026-10-30 13:27:58.631380964 +0000 UTC m=+1488.891619243"
Oct 30 13:27:59.596014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:59.595978    4291 generic.go:356] "Generic (PLEG): container finished" podID="f50c5c52-6fb8-4c5c-81b5-4242707a1da8" containerID="81c73958288f75e5fa45054cd6c7d6a34fb6641445f39afd9c4ed9ee5be4a9fc" exitCode=0
Oct 30 13:27:59.596414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:59.596031    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl" event={"ID":"f50c5c52-6fb8-4c5c-81b5-4242707a1da8","Type":"ContainerDied","Data":"81c73958288f75e5fa45054cd6c7d6a34fb6641445f39afd9c4ed9ee5be4a9fc"}
Oct 30 13:27:59.597749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:59.597723    4291 generic.go:356] "Generic (PLEG): container finished" podID="1f033bf8-00e6-4882-90fc-011782035d5e" containerID="14c5dfa24b3594516241b0cc2bb3ff68e30be925466d549a114a299b2f9c3255" exitCode=0
Oct 30 13:27:59.597885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:27:59.597778    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt" event={"ID":"1f033bf8-00e6-4882-90fc-011782035d5e","Type":"ContainerDied","Data":"14c5dfa24b3594516241b0cc2bb3ff68e30be925466d549a114a299b2f9c3255"}
Oct 30 13:28:00.797395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.797360    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:28:00.800522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.800503    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:28:00.894697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.894592    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-xks-ca\") pod \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") "
Oct 30 13:28:00.894697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.894686    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f033bf8-00e6-4882-90fc-011782035d5e-kube-api-access-fbpc5\" (UniqueName: \"kubernetes.io/projected/1f033bf8-00e6-4882-90fc-011782035d5e-kube-api-access-fbpc5\") pod \"1f033bf8-00e6-4882-90fc-011782035d5e\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") "
Oct 30 13:28:00.894934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.894839    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-xks-ca\") pod \"1f033bf8-00e6-4882-90fc-011782035d5e\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") "
Oct 30 13:28:00.894934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.894890    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-openshift-ca\") pod \"1f033bf8-00e6-4882-90fc-011782035d5e\" (UID: \"1f033bf8-00e6-4882-90fc-011782035d5e\") "
Oct 30 13:28:00.894934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.894929    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-openshift-ca\") pod \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") "
Oct 30 13:28:00.895107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.895070    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-kube-api-access-982gt\" (UniqueName: \"kubernetes.io/projected/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-kube-api-access-982gt\") pod \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\" (UID: \"f50c5c52-6fb8-4c5c-81b5-4242707a1da8\") "
Oct 30 13:28:00.895303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.895277    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-openshift-ca" pod "1f033bf8-00e6-4882-90fc-011782035d5e" (UID: "1f033bf8-00e6-4882-90fc-011782035d5e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:00.895745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.895723    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:00.895888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.895855    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-openshift-ca" pod "f50c5c52-6fb8-4c5c-81b5-4242707a1da8" (UID: "f50c5c52-6fb8-4c5c-81b5-4242707a1da8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:00.897972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.897908    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-kube-api-access-982gt" pod "f50c5c52-6fb8-4c5c-81b5-4242707a1da8" (UID: "f50c5c52-6fb8-4c5c-81b5-4242707a1da8"). InnerVolumeSpecName "kube-api-access-982gt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:00.897972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.897948    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f033bf8-00e6-4882-90fc-011782035d5e-kube-api-access-fbpc5" pod "1f033bf8-00e6-4882-90fc-011782035d5e" (UID: "1f033bf8-00e6-4882-90fc-011782035d5e"). InnerVolumeSpecName "kube-api-access-fbpc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:00.897972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.897960    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-xks-ca" pod "f50c5c52-6fb8-4c5c-81b5-4242707a1da8" (UID: "f50c5c52-6fb8-4c5c-81b5-4242707a1da8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:00.899325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.899298    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-xks-ca" pod "1f033bf8-00e6-4882-90fc-011782035d5e" (UID: "1f033bf8-00e6-4882-90fc-011782035d5e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:00.997516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.997480    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:00.997516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.997511    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbpc5\" (UniqueName: \"kubernetes.io/projected/1f033bf8-00e6-4882-90fc-011782035d5e-kube-api-access-fbpc5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:00.997516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.997522    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f033bf8-00e6-4882-90fc-011782035d5e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:00.997796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.997532    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:00.997796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:00.997541    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-982gt\" (UniqueName: \"kubernetes.io/projected/f50c5c52-6fb8-4c5c-81b5-4242707a1da8-kube-api-access-982gt\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:01.614048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:01.614004    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl" event={"ID":"f50c5c52-6fb8-4c5c-81b5-4242707a1da8","Type":"ContainerDied","Data":"04b732e07dce2f6f84e6c39a750950242162bcdad9db1d79dc0e20a5896c3f76"}
Oct 30 13:28:01.614048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:01.614032    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"
Oct 30 13:28:01.614310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:01.614106    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04b732e07dce2f6f84e6c39a750950242162bcdad9db1d79dc0e20a5896c3f76"
Oct 30 13:28:01.615500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:01.615467    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"
Oct 30 13:28:01.615617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:01.615521    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt" event={"ID":"1f033bf8-00e6-4882-90fc-011782035d5e","Type":"ContainerDied","Data":"fd0eecdab45518a6c2a5456f71916c125f13e7f7ca3222e537ffdfc0cec8c929"}
Oct 30 13:28:01.615617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:01.615561    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd0eecdab45518a6c2a5456f71916c125f13e7f7ca3222e537ffdfc0cec8c929"
Oct 30 13:28:02.735207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.735171    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"]
Oct 30 13:28:02.736075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.736054    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1f033bf8-00e6-4882-90fc-011782035d5e" containerName="revoke-keys"
Oct 30 13:28:02.736139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.736079    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f033bf8-00e6-4882-90fc-011782035d5e" containerName="revoke-keys"
Oct 30 13:28:02.736139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.736094    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f50c5c52-6fb8-4c5c-81b5-4242707a1da8" containerName="revoke-keys"
Oct 30 13:28:02.736139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.736103    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f50c5c52-6fb8-4c5c-81b5-4242707a1da8" containerName="revoke-keys"
Oct 30 13:28:02.736312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.736297    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f033bf8-00e6-4882-90fc-011782035d5e" containerName="revoke-keys"
Oct 30 13:28:02.736349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.736315    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f50c5c52-6fb8-4c5c-81b5-4242707a1da8" containerName="revoke-keys"
Oct 30 13:28:02.740380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.740362    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.760473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.760436    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"]
Oct 30 13:28:02.761082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.761056    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:02.761225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.761144    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:02.761409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.761388    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:02.761672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.761615    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:28:02.823103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.823070    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.823284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.823109    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjxwn\" (UniqueName: \"kubernetes.io/projected/25b4d15c-8109-4a74-ad03-e93cf6370992-kube-api-access-rjxwn\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.823338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.823312    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.924597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.924559    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.924796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.924779    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.924847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.924812    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjxwn\" (UniqueName: \"kubernetes.io/projected/25b4d15c-8109-4a74-ad03-e93cf6370992-kube-api-access-rjxwn\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.925348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.925328    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.927188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.927165    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-xks-ca\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:02.947058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:02.947031    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjxwn\" (UniqueName: \"kubernetes.io/projected/25b4d15c-8109-4a74-ad03-e93cf6370992-kube-api-access-rjxwn\") pod \"maas-api-revoke-sub-418f95ff-k8cwq\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:03.059923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:03.059827    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:03.411262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:03.411225    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"]
Oct 30 13:28:03.415061 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:28:03.415028    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b4d15c_8109_4a74_ad03_e93cf6370992.slice/crio-a3ba3e75c8e90ce93387e5f962f6a117e662da9931319771bf77ab7ffe4c7a5d WatchSource:0}: Error finding container a3ba3e75c8e90ce93387e5f962f6a117e662da9931319771bf77ab7ffe4c7a5d: Status 404 returned error can't find the container with id a3ba3e75c8e90ce93387e5f962f6a117e662da9931319771bf77ab7ffe4c7a5d
Oct 30 13:28:03.634811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:03.634771    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq" event={"ID":"25b4d15c-8109-4a74-ad03-e93cf6370992","Type":"ContainerStarted","Data":"a3ba3e75c8e90ce93387e5f962f6a117e662da9931319771bf77ab7ffe4c7a5d"}
Oct 30 13:28:04.642822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:04.642730    4291 generic.go:356] "Generic (PLEG): container finished" podID="25b4d15c-8109-4a74-ad03-e93cf6370992" containerID="3bb80961b4d8f729dd6378e70594fbe84d6274ee37deaa93bada8c27dc9a3eb1" exitCode=0
Oct 30 13:28:04.643204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:04.642815    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq" event={"ID":"25b4d15c-8109-4a74-ad03-e93cf6370992","Type":"ContainerDied","Data":"3bb80961b4d8f729dd6378e70594fbe84d6274ee37deaa93bada8c27dc9a3eb1"}
Oct 30 13:28:05.810055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.810029    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:05.863517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.863469    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b4d15c-8109-4a74-ad03-e93cf6370992-kube-api-access-rjxwn\" (UniqueName: \"kubernetes.io/projected/25b4d15c-8109-4a74-ad03-e93cf6370992-kube-api-access-rjxwn\") pod \"25b4d15c-8109-4a74-ad03-e93cf6370992\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") "
Oct 30 13:28:05.863711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.863622    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-openshift-ca\") pod \"25b4d15c-8109-4a74-ad03-e93cf6370992\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") "
Oct 30 13:28:05.863830 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.863813    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-xks-ca\") pod \"25b4d15c-8109-4a74-ad03-e93cf6370992\" (UID: \"25b4d15c-8109-4a74-ad03-e93cf6370992\") "
Oct 30 13:28:05.864057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.864021    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-openshift-ca" pod "25b4d15c-8109-4a74-ad03-e93cf6370992" (UID: "25b4d15c-8109-4a74-ad03-e93cf6370992"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:05.864493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.864461    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:05.865897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.865869    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b4d15c-8109-4a74-ad03-e93cf6370992-kube-api-access-rjxwn" pod "25b4d15c-8109-4a74-ad03-e93cf6370992" (UID: "25b4d15c-8109-4a74-ad03-e93cf6370992"). InnerVolumeSpecName "kube-api-access-rjxwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:05.866376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.866354    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-xks-ca" pod "25b4d15c-8109-4a74-ad03-e93cf6370992" (UID: "25b4d15c-8109-4a74-ad03-e93cf6370992"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:05.966275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.966191    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjxwn\" (UniqueName: \"kubernetes.io/projected/25b4d15c-8109-4a74-ad03-e93cf6370992-kube-api-access-rjxwn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:05.966275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:05.966223    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25b4d15c-8109-4a74-ad03-e93cf6370992-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:06.657739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:06.657707    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"
Oct 30 13:28:06.657919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:06.657707    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq" event={"ID":"25b4d15c-8109-4a74-ad03-e93cf6370992","Type":"ContainerDied","Data":"a3ba3e75c8e90ce93387e5f962f6a117e662da9931319771bf77ab7ffe4c7a5d"}
Oct 30 13:28:06.657919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:06.657867    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3ba3e75c8e90ce93387e5f962f6a117e662da9931319771bf77ab7ffe4c7a5d"
Oct 30 13:28:08.224334 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:28:08.224277    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:28:11.059720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.059682    4291 scope.go:122] "RemoveContainer" containerID="83542fc9082ed6e238894c6d20bee8b23c105573e67cde3d2e05890af298ede3"
Oct 30 13:28:11.070174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.070153    4291 scope.go:122] "RemoveContainer" containerID="4d3aef48f715f0fd937d6de32b7d3553b1d40d825e9a804ea29368365c78fa67"
Oct 30 13:28:11.079854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.079827    4291 scope.go:122] "RemoveContainer" containerID="f9d749396163160285f17a560d6b1f31fcd4e1083861826ccc407d65d3345007"
Oct 30 13:28:11.925975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.925941    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"]
Oct 30 13:28:11.926505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.926491    4291 cpu_manager.go:414] "containerMap: removing container" podUID="25b4d15c-8109-4a74-ad03-e93cf6370992" containerName="revoke-keys"
Oct 30 13:28:11.926549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.926507    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b4d15c-8109-4a74-ad03-e93cf6370992" containerName="revoke-keys"
Oct 30 13:28:11.926595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.926586    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b4d15c-8109-4a74-ad03-e93cf6370992" containerName="revoke-keys"
Oct 30 13:28:11.931407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.931387    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:11.947449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.947419    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"]
Oct 30 13:28:11.953917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.953882    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:11.953917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.953883    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:28:11.954156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.954023    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:11.957554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:11.957531    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:12.048259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.048213    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wfbw\" (UniqueName: \"kubernetes.io/projected/be6b51a7-9805-4092-adab-1eb477500eb8-kube-api-access-8wfbw\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.048439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.048368    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.048439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.048393    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.149996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.149962    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.149996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.150001    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.150540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.150145    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wfbw\" (UniqueName: \"kubernetes.io/projected/be6b51a7-9805-4092-adab-1eb477500eb8-kube-api-access-8wfbw\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.150790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.150769    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.152778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.152751    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.169767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.169739    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wfbw\" (UniqueName: \"kubernetes.io/projected/be6b51a7-9805-4092-adab-1eb477500eb8-kube-api-access-8wfbw\") pod \"maas-api-revoke-sub-273ad47d-84m2s\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.251233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.251150    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:12.536715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.536613    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:28:12.563227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.563199    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"]
Oct 30 13:28:12.563466 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:28:12.563437    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6b51a7_9805_4092_adab_1eb477500eb8.slice/crio-d619a0675c710494f3585af9864149331006cad57cc15af4092eed8149e6d018 WatchSource:0}: Error finding container d619a0675c710494f3585af9864149331006cad57cc15af4092eed8149e6d018: Status 404 returned error can't find the container with id d619a0675c710494f3585af9864149331006cad57cc15af4092eed8149e6d018
Oct 30 13:28:12.700058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:12.700020    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s" event={"ID":"be6b51a7-9805-4092-adab-1eb477500eb8","Type":"ContainerStarted","Data":"d619a0675c710494f3585af9864149331006cad57cc15af4092eed8149e6d018"}
Oct 30 13:28:13.709760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:13.709667    4291 generic.go:356] "Generic (PLEG): container finished" podID="be6b51a7-9805-4092-adab-1eb477500eb8" containerID="6f8cb84ee78e23f83b288200f154b5c368dfe25579dcd5524493e4475479ce5a" exitCode=0
Oct 30 13:28:13.710164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:13.709812    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s" event={"ID":"be6b51a7-9805-4092-adab-1eb477500eb8","Type":"ContainerDied","Data":"6f8cb84ee78e23f83b288200f154b5c368dfe25579dcd5524493e4475479ce5a"}
Oct 30 13:28:15.419996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.419950    4291 patch_prober.go:29] interesting pod/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:28:15.420563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.420533    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:28:15.581828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.581490    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:15.796499 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.796400    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"]
Oct 30 13:28:15.797584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.797555    4291 cpu_manager.go:414] "containerMap: removing container" podUID="be6b51a7-9805-4092-adab-1eb477500eb8" containerName="revoke-keys"
Oct 30 13:28:15.802296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.802262    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6b51a7-9805-4092-adab-1eb477500eb8" containerName="revoke-keys"
Oct 30 13:28:15.802748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.802728    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6b51a7-9805-4092-adab-1eb477500eb8" containerName="revoke-keys"
Oct 30 13:28:15.812472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.812419    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6b51a7-9805-4092-adab-1eb477500eb8-kube-api-access-8wfbw\" (UniqueName: \"kubernetes.io/projected/be6b51a7-9805-4092-adab-1eb477500eb8-kube-api-access-8wfbw\") pod \"be6b51a7-9805-4092-adab-1eb477500eb8\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") "
Oct 30 13:28:15.812766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.812742    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-openshift-ca\") pod \"be6b51a7-9805-4092-adab-1eb477500eb8\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") "
Oct 30 13:28:15.813215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.813180    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-xks-ca\") pod \"be6b51a7-9805-4092-adab-1eb477500eb8\" (UID: \"be6b51a7-9805-4092-adab-1eb477500eb8\") "
Oct 30 13:28:15.823477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.821919    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-openshift-ca" pod "be6b51a7-9805-4092-adab-1eb477500eb8" (UID: "be6b51a7-9805-4092-adab-1eb477500eb8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:15.855253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.855024    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"
Oct 30 13:28:15.875842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.875800    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:15.892696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.890967    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-xks-ca" pod "be6b51a7-9805-4092-adab-1eb477500eb8" (UID: "be6b51a7-9805-4092-adab-1eb477500eb8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:15.911264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.911231    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"]
Oct 30 13:28:15.911613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.911587    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s" event={"ID":"be6b51a7-9805-4092-adab-1eb477500eb8","Type":"ContainerDied","Data":"d619a0675c710494f3585af9864149331006cad57cc15af4092eed8149e6d018"}
Oct 30 13:28:15.911798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.911781    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d619a0675c710494f3585af9864149331006cad57cc15af4092eed8149e6d018"
Oct 30 13:28:15.925673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:15.919111    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6b51a7-9805-4092-adab-1eb477500eb8-kube-api-access-8wfbw" pod "be6b51a7-9805-4092-adab-1eb477500eb8" (UID: "be6b51a7-9805-4092-adab-1eb477500eb8"). InnerVolumeSpecName "kube-api-access-8wfbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:16.038016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.027244    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:16.038016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.027281    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6b51a7-9805-4092-adab-1eb477500eb8-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:16.133975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.133931    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.134267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.134244    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.134518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.134506    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8vd2\" (UniqueName: \"kubernetes.io/projected/7052ec8a-cdd4-496f-b2c6-d29275e1053e-kube-api-access-c8vd2\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.135939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.135920    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wfbw\" (UniqueName: \"kubernetes.io/projected/be6b51a7-9805-4092-adab-1eb477500eb8-kube-api-access-8wfbw\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:16.242344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.237244    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.242344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.237340    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.242344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.237421    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8vd2\" (UniqueName: \"kubernetes.io/projected/7052ec8a-cdd4-496f-b2c6-d29275e1053e-kube-api-access-c8vd2\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.242344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.239479    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.243088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.243026    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.290705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.290366    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8vd2\" (UniqueName: \"kubernetes.io/projected/7052ec8a-cdd4-496f-b2c6-d29275e1053e-kube-api-access-c8vd2\") pod \"maas-api-revoke-sub-6213b77f-p2ntd\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.292088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.291697    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:16.655000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.647218    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"]
Oct 30 13:28:16.863292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:16.863258    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd" event={"ID":"7052ec8a-cdd4-496f-b2c6-d29275e1053e","Type":"ContainerStarted","Data":"7c078a7f385a553df8e854e45d231556b19b1febd681530176292a851445f421"}
Oct 30 13:28:17.876319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:17.876286    4291 generic.go:356] "Generic (PLEG): container finished" podID="7052ec8a-cdd4-496f-b2c6-d29275e1053e" containerID="65d319539b21ed5355e98355725c0ad71edadf259acc9dcfc1d1a63e6518ad27" exitCode=0
Oct 30 13:28:17.880398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:17.880363    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd" event={"ID":"7052ec8a-cdd4-496f-b2c6-d29275e1053e","Type":"ContainerDied","Data":"65d319539b21ed5355e98355725c0ad71edadf259acc9dcfc1d1a63e6518ad27"}
Oct 30 13:28:19.059612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.059584    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:19.098973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.098922    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7052ec8a-cdd4-496f-b2c6-d29275e1053e-kube-api-access-c8vd2\" (UniqueName: \"kubernetes.io/projected/7052ec8a-cdd4-496f-b2c6-d29275e1053e-kube-api-access-c8vd2\") pod \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") "
Oct 30 13:28:19.099168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.099064    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-openshift-ca\") pod \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") "
Oct 30 13:28:19.099168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.099157    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-xks-ca\") pod \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\" (UID: \"7052ec8a-cdd4-496f-b2c6-d29275e1053e\") "
Oct 30 13:28:19.099729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.099624    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-openshift-ca" pod "7052ec8a-cdd4-496f-b2c6-d29275e1053e" (UID: "7052ec8a-cdd4-496f-b2c6-d29275e1053e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:19.099857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.099745    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:19.101896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.101866    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7052ec8a-cdd4-496f-b2c6-d29275e1053e-kube-api-access-c8vd2" pod "7052ec8a-cdd4-496f-b2c6-d29275e1053e" (UID: "7052ec8a-cdd4-496f-b2c6-d29275e1053e"). InnerVolumeSpecName "kube-api-access-c8vd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:19.102046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.101938    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-xks-ca" pod "7052ec8a-cdd4-496f-b2c6-d29275e1053e" (UID: "7052ec8a-cdd4-496f-b2c6-d29275e1053e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:19.200918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.200821    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8vd2\" (UniqueName: \"kubernetes.io/projected/7052ec8a-cdd4-496f-b2c6-d29275e1053e-kube-api-access-c8vd2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:19.200918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.200855    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7052ec8a-cdd4-496f-b2c6-d29275e1053e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:19.894572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.894544    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"
Oct 30 13:28:19.894760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.894579    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd" event={"ID":"7052ec8a-cdd4-496f-b2c6-d29275e1053e","Type":"ContainerDied","Data":"7c078a7f385a553df8e854e45d231556b19b1febd681530176292a851445f421"}
Oct 30 13:28:19.894760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:19.894650    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c078a7f385a553df8e854e45d231556b19b1febd681530176292a851445f421"
Oct 30 13:28:21.591685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:21.591631    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"]
Oct 30 13:28:21.601877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:21.601833    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-b35c0f-cdzvt"]
Oct 30 13:28:21.898242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:21.898199    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f033bf8-00e6-4882-90fc-011782035d5e" path="/var/lib/kubelet/pods/1f033bf8-00e6-4882-90fc-011782035d5e/volumes"
Oct 30 13:28:25.375418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:25.375379    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"]
Oct 30 13:28:25.375846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:25.375600    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy" containerID="cri-o://5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222" gracePeriod=30
Oct 30 13:28:25.389711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:25.389673    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"]
Oct 30 13:28:25.390225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:25.390165    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy" containerID="cri-o://c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c" gracePeriod=30
Oct 30 13:28:27.349215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:27.349167    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"]
Oct 30 13:28:27.361587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:27.361550    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-5cee02-tr5hx"]
Oct 30 13:28:27.879074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:27.879039    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94f183c2-fa1a-4620-9812-0898de7e146c" path="/var/lib/kubelet/pods/94f183c2-fa1a-4620-9812-0898de7e146c/volumes"
Oct 30 13:28:29.233491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.233458    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"]
Oct 30 13:28:29.234014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.233999    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7052ec8a-cdd4-496f-b2c6-d29275e1053e" containerName="revoke-keys"
Oct 30 13:28:29.234059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.234015    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7052ec8a-cdd4-496f-b2c6-d29275e1053e" containerName="revoke-keys"
Oct 30 13:28:29.234115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.234106    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7052ec8a-cdd4-496f-b2c6-d29275e1053e" containerName="revoke-keys"
Oct 30 13:28:29.238579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.238562    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.254271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.254239    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"]
Oct 30 13:28:29.261916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.261875    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:29.262061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.261942    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:29.304352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.304316    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:29.304533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.304394    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:28:29.393466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.393424    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96x5\" (UniqueName: \"kubernetes.io/projected/d975de3a-7140-4aaf-8d85-3fa69b46540d-kube-api-access-c96x5\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.393693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.393573    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.393693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.393673    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.495157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.495069    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.495340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.495165    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.495340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.495292    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96x5\" (UniqueName: \"kubernetes.io/projected/d975de3a-7140-4aaf-8d85-3fa69b46540d-kube-api-access-c96x5\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.495861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.495836    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.497593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.497574    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.514319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.514290    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96x5\" (UniqueName: \"kubernetes.io/projected/d975de3a-7140-4aaf-8d85-3fa69b46540d-kube-api-access-c96x5\") pod \"maas-api-revoke-sub-471184b7-692h7\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.555339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.555303    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:29.862511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.862479    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"]
Oct 30 13:28:29.863812 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:28:29.863786    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd975de3a_7140_4aaf_8d85_3fa69b46540d.slice/crio-e55db0210cdfd76782ca49739636704b29f34bfc256ac98e952fd8bb39e02ab1 WatchSource:0}: Error finding container e55db0210cdfd76782ca49739636704b29f34bfc256ac98e952fd8bb39e02ab1: Status 404 returned error can't find the container with id e55db0210cdfd76782ca49739636704b29f34bfc256ac98e952fd8bb39e02ab1
Oct 30 13:28:29.990019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:29.989981    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7" event={"ID":"d975de3a-7140-4aaf-8d85-3fa69b46540d","Type":"ContainerStarted","Data":"e55db0210cdfd76782ca49739636704b29f34bfc256ac98e952fd8bb39e02ab1"}
Oct 30 13:28:30.310333 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.310298    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"]
Oct 30 13:28:30.310791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.310501    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy" containerID="cri-o://118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f" gracePeriod=30
Oct 30 13:28:30.403200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.403171    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"]
Oct 30 13:28:30.407421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.407403    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.429288 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.429258    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"]
Oct 30 13:28:30.508550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.508521    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.509261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.508982    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/14e6452f-65c5-4d47-beb9-b821e19eeeae-kube-api-access-pqd6z\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.509261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.509054    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.610574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.610531    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/14e6452f-65c5-4d47-beb9-b821e19eeeae-kube-api-access-pqd6z\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.610835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.610608    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.610835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.610738    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.611407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.611380    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.613260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.613230    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.632808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.632014    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/14e6452f-65c5-4d47-beb9-b821e19eeeae-kube-api-access-pqd6z\") pod \"maas-api-revoke-sub-c7284524-pmd2g\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.685244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.685165    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:28:30.704702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.704678    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:28:30.711386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711360    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a8157e3-282b-444e-ad9e-52c201f99492-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8157e3-282b-444e-ad9e-52c201f99492-istio-podinfo\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.711527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711412    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-envoy\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.711527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711446    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-socket\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.711527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711476    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-certs\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.711527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711510    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-credential-socket\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.711753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711592    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-kube-api-access-bscw4\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-kube-api-access-bscw4\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.711753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711620    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-istio-token\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-istio-token\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.711753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711695    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-envoy\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.711753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711740    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-socket" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.711926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711753    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-credential-socket\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.711926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711773    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-certs" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.711926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711818    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-data\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.711926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711857    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-credential-socket" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.711926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711883    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-kube-api-access-cgsmj\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-kube-api-access-cgsmj\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.712223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711939    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-socket\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.712223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.711972    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-certs\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.712223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712065    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istiod-ca-cert\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.712223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712110    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-podinfo\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.712223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712172    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-token\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-token\") pod \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\" (UID: \"90eccdf7-bf8b-45d5-8302-8e7c88eca050\") "
Oct 30 13:28:30.712223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712201    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-credential-socket" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.712573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712218    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-data" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.712573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712306    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a8157e3-282b-444e-ad9e-52c201f99492-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8157e3-282b-444e-ad9e-52c201f99492-istiod-ca-cert\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.712573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712359    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-data\") pod \"1a8157e3-282b-444e-ad9e-52c201f99492\" (UID: \"1a8157e3-282b-444e-ad9e-52c201f99492\") "
Oct 30 13:28:30.712573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.712443    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-certs" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.713072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713047    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.713160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713078    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.713160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713092    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.713160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713106    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.713160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713121    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.713160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713136    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.713452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713336    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-data" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.713820 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713792    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-socket" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.713918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713875    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istiod-ca-cert" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:30.714020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.713997    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a8157e3-282b-444e-ad9e-52c201f99492-istiod-ca-cert" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:30.714249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.714207    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-istio-token" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:30.714407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.714383    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a8157e3-282b-444e-ad9e-52c201f99492-istio-podinfo" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:30.714555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.714533    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-token" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:30.714555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.714549    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-envoy" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.714733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.714562    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-kube-api-access-bscw4" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "kube-api-access-bscw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:30.714852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.714837    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-kube-api-access-cgsmj" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "kube-api-access-cgsmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:30.715396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.715381    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-envoy" pod "1a8157e3-282b-444e-ad9e-52c201f99492" (UID: "1a8157e3-282b-444e-ad9e-52c201f99492"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:30.715912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.715898    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-podinfo" pod "90eccdf7-bf8b-45d5-8302-8e7c88eca050" (UID: "90eccdf7-bf8b-45d5-8302-8e7c88eca050"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:30.788558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.788323    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:30.813935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813907    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bscw4\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-kube-api-access-bscw4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.813935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813933    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.813935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813943    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813951    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgsmj\" (UniqueName: \"kubernetes.io/projected/1a8157e3-282b-444e-ad9e-52c201f99492-kube-api-access-cgsmj\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813960    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813969    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813977    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813985    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813992    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a8157e3-282b-444e-ad9e-52c201f99492-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.813999    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a8157e3-282b-444e-ad9e-52c201f99492-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.814005    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a8157e3-282b-444e-ad9e-52c201f99492-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:30.814141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:30.814013    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90eccdf7-bf8b-45d5-8302-8e7c88eca050-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:31.001268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.001164    4291 generic.go:356] "Generic (PLEG): container finished" podID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerID="5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222" exitCode=0
Oct 30 13:28:31.001268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.001240    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"
Oct 30 13:28:31.001268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.001246    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" event={"ID":"90eccdf7-bf8b-45d5-8302-8e7c88eca050","Type":"ContainerDied","Data":"5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222"}
Oct 30 13:28:31.001553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.001313    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc" event={"ID":"90eccdf7-bf8b-45d5-8302-8e7c88eca050","Type":"ContainerDied","Data":"976735cd5546a55388147226bb887a289a1fb04836c64d04d818cd81c665781a"}
Oct 30 13:28:31.001553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.001346    4291 scope.go:122] "RemoveContainer" containerID="5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222"
Oct 30 13:28:31.003683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.003657    4291 generic.go:356] "Generic (PLEG): container finished" podID="d975de3a-7140-4aaf-8d85-3fa69b46540d" containerID="e2e525ec7293a1b24df994f0741d6f9967964d37ff1e92cab352bcba64d4dc10" exitCode=0
Oct 30 13:28:31.003827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.003686    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7" event={"ID":"d975de3a-7140-4aaf-8d85-3fa69b46540d","Type":"ContainerDied","Data":"e2e525ec7293a1b24df994f0741d6f9967964d37ff1e92cab352bcba64d4dc10"}
Oct 30 13:28:31.005289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.005268    4291 generic.go:356] "Generic (PLEG): container finished" podID="1a8157e3-282b-444e-ad9e-52c201f99492" containerID="c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c" exitCode=0
Oct 30 13:28:31.005388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.005310    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" event={"ID":"1a8157e3-282b-444e-ad9e-52c201f99492","Type":"ContainerDied","Data":"c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c"}
Oct 30 13:28:31.005388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.005324    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"
Oct 30 13:28:31.005388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.005353    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b" event={"ID":"1a8157e3-282b-444e-ad9e-52c201f99492","Type":"ContainerDied","Data":"2ed6f2bdff2d8dac481244a4cf8f4a82990675c4654624b86657333a8d6039f6"}
Oct 30 13:28:31.018948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.018930    4291 scope.go:122] "RemoveContainer" containerID="5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222"
Oct 30 13:28:31.019311 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:28:31.019288    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222\": container with ID starting with 5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222 not found: ID does not exist" containerID="5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222"
Oct 30 13:28:31.019372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.019322    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222"} err="failed to get container status \"5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222\": rpc error: code = NotFound desc = could not find container \"5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222\": container with ID starting with 5baaf4f134432db4a7784e2244d1e6d97aff54326e660206e8d499d802445222 not found: ID does not exist"
Oct 30 13:28:31.019372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.019339    4291 scope.go:122] "RemoveContainer" containerID="c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c"
Oct 30 13:28:31.029762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.029740    4291 scope.go:122] "RemoveContainer" containerID="c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c"
Oct 30 13:28:31.030053 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:28:31.030024    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c\": container with ID starting with c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c not found: ID does not exist" containerID="c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c"
Oct 30 13:28:31.030140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.030062    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c"} err="failed to get container status \"c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c\": rpc error: code = NotFound desc = could not find container \"c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c\": container with ID starting with c06fe395e56835bc1de1c89c77dcf50b454431412e58828f33986c5fb243c11c not found: ID does not exist"
Oct 30 13:28:31.097601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.097571    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"]
Oct 30 13:28:31.098498 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:28:31.098467    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e6452f_65c5_4d47_beb9_b821e19eeeae.slice/crio-e4442991d0eb19c0fe35d015ebd04a81225f5bd196047e731964a6c1a063d2b9 WatchSource:0}: Error finding container e4442991d0eb19c0fe35d015ebd04a81225f5bd196047e731964a6c1a063d2b9: Status 404 returned error can't find the container with id e4442991d0eb19c0fe35d015ebd04a81225f5bd196047e731964a6c1a063d2b9
Oct 30 13:28:31.106172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.106150    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"]
Oct 30 13:28:31.109821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.109773    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-j7pdc"]
Oct 30 13:28:31.142786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.142754    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"]
Oct 30 13:28:31.146654 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.146606    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-65n4b"]
Oct 30 13:28:31.879403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.879367    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" path="/var/lib/kubelet/pods/1a8157e3-282b-444e-ad9e-52c201f99492/volumes"
Oct 30 13:28:31.879908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:31.879892    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" path="/var/lib/kubelet/pods/90eccdf7-bf8b-45d5-8302-8e7c88eca050/volumes"
Oct 30 13:28:32.016513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.016475    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g" event={"ID":"14e6452f-65c5-4d47-beb9-b821e19eeeae","Type":"ContainerStarted","Data":"f8e82d9a0a8aa8d42bf58099141637ac1fd06060cde55320160c3fa8c31956c8"}
Oct 30 13:28:32.016737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.016538    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g" event={"ID":"14e6452f-65c5-4d47-beb9-b821e19eeeae","Type":"ContainerStarted","Data":"e4442991d0eb19c0fe35d015ebd04a81225f5bd196047e731964a6c1a063d2b9"}
Oct 30 13:28:32.061842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.061778    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g" podStartSLOduration=2.061753743 podStartE2EDuration="2.061753743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:32.053820641 +0000 UTC m=+1522.314058896" watchObservedRunningTime="2026-10-30 13:28:32.061753743 +0000 UTC m=+1522.321991978"
Oct 30 13:28:32.182483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.182453    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:32.231595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.231564    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-xks-ca\") pod \"d975de3a-7140-4aaf-8d85-3fa69b46540d\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") "
Oct 30 13:28:32.231782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.231701    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d975de3a-7140-4aaf-8d85-3fa69b46540d-kube-api-access-c96x5\" (UniqueName: \"kubernetes.io/projected/d975de3a-7140-4aaf-8d85-3fa69b46540d-kube-api-access-c96x5\") pod \"d975de3a-7140-4aaf-8d85-3fa69b46540d\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") "
Oct 30 13:28:32.231890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.231872    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-openshift-ca\") pod \"d975de3a-7140-4aaf-8d85-3fa69b46540d\" (UID: \"d975de3a-7140-4aaf-8d85-3fa69b46540d\") "
Oct 30 13:28:32.232228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.232204    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-openshift-ca" pod "d975de3a-7140-4aaf-8d85-3fa69b46540d" (UID: "d975de3a-7140-4aaf-8d85-3fa69b46540d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:32.232427 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.232411    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:32.233758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.233730    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-xks-ca" pod "d975de3a-7140-4aaf-8d85-3fa69b46540d" (UID: "d975de3a-7140-4aaf-8d85-3fa69b46540d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:32.233872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.233852    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d975de3a-7140-4aaf-8d85-3fa69b46540d-kube-api-access-c96x5" pod "d975de3a-7140-4aaf-8d85-3fa69b46540d" (UID: "d975de3a-7140-4aaf-8d85-3fa69b46540d"). InnerVolumeSpecName "kube-api-access-c96x5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:32.334117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.334080    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d975de3a-7140-4aaf-8d85-3fa69b46540d-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:32.334117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:32.334115    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96x5\" (UniqueName: \"kubernetes.io/projected/d975de3a-7140-4aaf-8d85-3fa69b46540d-kube-api-access-c96x5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:33.026247 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:33.026209    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"
Oct 30 13:28:33.026247 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:33.026222    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7" event={"ID":"d975de3a-7140-4aaf-8d85-3fa69b46540d","Type":"ContainerDied","Data":"e55db0210cdfd76782ca49739636704b29f34bfc256ac98e952fd8bb39e02ab1"}
Oct 30 13:28:33.026870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:33.026282    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e55db0210cdfd76782ca49739636704b29f34bfc256ac98e952fd8bb39e02ab1"
Oct 30 13:28:33.028690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:33.028661    4291 generic.go:356] "Generic (PLEG): container finished" podID="14e6452f-65c5-4d47-beb9-b821e19eeeae" containerID="f8e82d9a0a8aa8d42bf58099141637ac1fd06060cde55320160c3fa8c31956c8" exitCode=0
Oct 30 13:28:33.028818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:33.028803    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g" event={"ID":"14e6452f-65c5-4d47-beb9-b821e19eeeae","Type":"ContainerDied","Data":"f8e82d9a0a8aa8d42bf58099141637ac1fd06060cde55320160c3fa8c31956c8"}
Oct 30 13:28:34.199812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.199780    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:34.258123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.258087    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-openshift-ca\") pod \"14e6452f-65c5-4d47-beb9-b821e19eeeae\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") "
Oct 30 13:28:34.258335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.258136    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e6452f-65c5-4d47-beb9-b821e19eeeae-kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/14e6452f-65c5-4d47-beb9-b821e19eeeae-kube-api-access-pqd6z\") pod \"14e6452f-65c5-4d47-beb9-b821e19eeeae\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") "
Oct 30 13:28:34.258335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.258234    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-xks-ca\") pod \"14e6452f-65c5-4d47-beb9-b821e19eeeae\" (UID: \"14e6452f-65c5-4d47-beb9-b821e19eeeae\") "
Oct 30 13:28:34.258539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.258515    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-openshift-ca" pod "14e6452f-65c5-4d47-beb9-b821e19eeeae" (UID: "14e6452f-65c5-4d47-beb9-b821e19eeeae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:34.258777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.258758    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:34.260338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.260309    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e6452f-65c5-4d47-beb9-b821e19eeeae-kube-api-access-pqd6z" pod "14e6452f-65c5-4d47-beb9-b821e19eeeae" (UID: "14e6452f-65c5-4d47-beb9-b821e19eeeae"). InnerVolumeSpecName "kube-api-access-pqd6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:34.260763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.260742    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-xks-ca" pod "14e6452f-65c5-4d47-beb9-b821e19eeeae" (UID: "14e6452f-65c5-4d47-beb9-b821e19eeeae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:34.360398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.360356    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqd6z\" (UniqueName: \"kubernetes.io/projected/14e6452f-65c5-4d47-beb9-b821e19eeeae-kube-api-access-pqd6z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:34.360398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:34.360389    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/14e6452f-65c5-4d47-beb9-b821e19eeeae-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.044289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.044261    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"
Oct 30 13:28:35.044472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.044293    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g" event={"ID":"14e6452f-65c5-4d47-beb9-b821e19eeeae","Type":"ContainerDied","Data":"e4442991d0eb19c0fe35d015ebd04a81225f5bd196047e731964a6c1a063d2b9"}
Oct 30 13:28:35.044472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.044343    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4442991d0eb19c0fe35d015ebd04a81225f5bd196047e731964a6c1a063d2b9"
Oct 30 13:28:35.587054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.586990    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.677610    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-token\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-token\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.677674    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-kube-api-access-srzmg\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-kube-api-access-srzmg\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.677789    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-socket\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.677843    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istiod-ca-cert\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.677903    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-certs\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.677964    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-credential-socket\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.678036    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-podinfo\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.678053    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-data\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.678696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.678096    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-envoy\") pod \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\" (UID: \"8bb2aeb5-62a0-439d-818d-1bb8c9507942\") "
Oct 30 13:28:35.679291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.679037    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istiod-ca-cert" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:35.679291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.679140    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-credential-socket" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:35.679431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.679410    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-certs" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:35.679843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.679820    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-data" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:35.680032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.679979    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-socket" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:35.681676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.681515    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-envoy" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:28:35.681676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.681611    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-podinfo" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:28:35.682004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.681978    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-kube-api-access-srzmg" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "kube-api-access-srzmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:35.682085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.682063    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-token" pod "8bb2aeb5-62a0-439d-818d-1bb8c9507942" (UID: "8bb2aeb5-62a0-439d-818d-1bb8c9507942"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:35.779717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779629    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779705    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srzmg\" (UniqueName: \"kubernetes.io/projected/8bb2aeb5-62a0-439d-818d-1bb8c9507942-kube-api-access-srzmg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779721    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779735    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779747    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779757    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779768    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779779    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:35.779978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:35.779789    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8bb2aeb5-62a0-439d-818d-1bb8c9507942-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:36.053019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.052982    4291 generic.go:356] "Generic (PLEG): container finished" podID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerID="118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f" exitCode=0
Oct 30 13:28:36.053019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.053022    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" event={"ID":"8bb2aeb5-62a0-439d-818d-1bb8c9507942","Type":"ContainerDied","Data":"118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f"}
Oct 30 13:28:36.053240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.053034    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"
Oct 30 13:28:36.053240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.053070    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h" event={"ID":"8bb2aeb5-62a0-439d-818d-1bb8c9507942","Type":"ContainerDied","Data":"1cbeadd6f7e8cc8cfa31ba3fbbf925a8729e943b4184fcd2a5bc4bd1e6833f65"}
Oct 30 13:28:36.053240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.053092    4291 scope.go:122] "RemoveContainer" containerID="118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f"
Oct 30 13:28:36.063253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.063224    4291 scope.go:122] "RemoveContainer" containerID="118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f"
Oct 30 13:28:36.063560 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:28:36.063539    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f\": container with ID starting with 118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f not found: ID does not exist" containerID="118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f"
Oct 30 13:28:36.063624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.063567    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f"} err="failed to get container status \"118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f\": rpc error: code = NotFound desc = could not find container \"118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f\": container with ID starting with 118cb06258826f440230d4ae78e56f7d68f89a7fd931d6d1978ea4aefb14777f not found: ID does not exist"
Oct 30 13:28:36.119771 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.119734    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"]
Oct 30 13:28:36.131557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:36.131528    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-tdl7h"]
Oct 30 13:28:37.934547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:37.934502    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" path="/var/lib/kubelet/pods/8bb2aeb5-62a0-439d-818d-1bb8c9507942/volumes"
Oct 30 13:28:38.343714 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:28:38.334872    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:28:39.210570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.210531    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"]
Oct 30 13:28:39.211181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211163    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy"
Oct 30 13:28:39.211181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211181    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211194    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211200    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211208    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d975de3a-7140-4aaf-8d85-3fa69b46540d" containerName="revoke-keys"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211214    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d975de3a-7140-4aaf-8d85-3fa69b46540d" containerName="revoke-keys"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211226    4291 cpu_manager.go:414] "containerMap: removing container" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211231    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211237    4291 cpu_manager.go:414] "containerMap: removing container" podUID="14e6452f-65c5-4d47-beb9-b821e19eeeae" containerName="revoke-keys"
Oct 30 13:28:39.211292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211242    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e6452f-65c5-4d47-beb9-b821e19eeeae" containerName="revoke-keys"
Oct 30 13:28:39.211710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211330    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="90eccdf7-bf8b-45d5-8302-8e7c88eca050" containerName="istio-proxy"
Oct 30 13:28:39.211710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211337    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e6452f-65c5-4d47-beb9-b821e19eeeae" containerName="revoke-keys"
Oct 30 13:28:39.211710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211348    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bb2aeb5-62a0-439d-818d-1bb8c9507942" containerName="istio-proxy"
Oct 30 13:28:39.211710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211354    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a8157e3-282b-444e-ad9e-52c201f99492" containerName="istio-proxy"
Oct 30 13:28:39.211710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.211360    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d975de3a-7140-4aaf-8d85-3fa69b46540d" containerName="revoke-keys"
Oct 30 13:28:39.216670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.216630    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.231315 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.231276    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"]
Oct 30 13:28:39.237729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.237697    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:39.237977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.237951    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:39.237977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.237976    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:39.242845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.242817    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:28:39.270310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.270267    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.270500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.270384    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjdk5\" (UniqueName: \"kubernetes.io/projected/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-kube-api-access-gjdk5\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.270500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.270447    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.372272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.372234    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.372462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.372326    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjdk5\" (UniqueName: \"kubernetes.io/projected/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-kube-api-access-gjdk5\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.372462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.372399    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.373174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.373149    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.374909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.374888    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.391301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.391272    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjdk5\" (UniqueName: \"kubernetes.io/projected/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-kube-api-access-gjdk5\") pod \"maas-api-revoke-sub-df7a53b6-dkl6c\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.533334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.533251    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:39.837060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:39.837029    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"]
Oct 30 13:28:39.838569 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:28:39.838546    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53cdf8b7_d83b_4bc0_a435_d31d9ae05f9f.slice/crio-175479300c93024a62a88c8c3a48f217948f58de99ec85609c0358e50e544b55 WatchSource:0}: Error finding container 175479300c93024a62a88c8c3a48f217948f58de99ec85609c0358e50e544b55: Status 404 returned error can't find the container with id 175479300c93024a62a88c8c3a48f217948f58de99ec85609c0358e50e544b55
Oct 30 13:28:40.105097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:40.105058    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c" event={"ID":"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f","Type":"ContainerStarted","Data":"175479300c93024a62a88c8c3a48f217948f58de99ec85609c0358e50e544b55"}
Oct 30 13:28:41.113526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:41.113490    4291 generic.go:356] "Generic (PLEG): container finished" podID="53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" containerID="8bb28d0dc26797732c673de90c9768af82f28ba71b878db50e8134d70e685216" exitCode=0
Oct 30 13:28:41.113931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:41.113576    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c" event={"ID":"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f","Type":"ContainerDied","Data":"8bb28d0dc26797732c673de90c9768af82f28ba71b878db50e8134d70e685216"}
Oct 30 13:28:42.489758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.489733    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:42.628588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.628544    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-kube-api-access-gjdk5\" (UniqueName: \"kubernetes.io/projected/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-kube-api-access-gjdk5\") pod \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") "
Oct 30 13:28:42.628795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.628674    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-xks-ca\") pod \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") "
Oct 30 13:28:42.628795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.628713    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-openshift-ca\") pod \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\" (UID: \"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f\") "
Oct 30 13:28:42.629123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.629099    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-openshift-ca" pod "53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" (UID: "53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:42.630702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.630678    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-xks-ca" pod "53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" (UID: "53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:42.630839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.630747    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-kube-api-access-gjdk5" pod "53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" (UID: "53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f"). InnerVolumeSpecName "kube-api-access-gjdk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:42.729798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.729762    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:42.729798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.729790    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjdk5\" (UniqueName: \"kubernetes.io/projected/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-kube-api-access-gjdk5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:42.729798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:42.729802    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:43.342348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.342316    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"
Oct 30 13:28:43.342348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.342340    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c" event={"ID":"53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f","Type":"ContainerDied","Data":"175479300c93024a62a88c8c3a48f217948f58de99ec85609c0358e50e544b55"}
Oct 30 13:28:43.342589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.342401    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="175479300c93024a62a88c8c3a48f217948f58de99ec85609c0358e50e544b55"
Oct 30 13:28:43.785400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.785368    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"]
Oct 30 13:28:43.786113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.786091    4291 cpu_manager.go:414] "containerMap: removing container" podUID="53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" containerName="revoke-keys"
Oct 30 13:28:43.786226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.786115    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" containerName="revoke-keys"
Oct 30 13:28:43.786310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.786296    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" containerName="revoke-keys"
Oct 30 13:28:43.789858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.789837    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.805885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.805855    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"]
Oct 30 13:28:43.813881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.813852    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:43.813881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.813879    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:43.814098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.813889    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:43.815438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.815419    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:28:43.840590 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.840553    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.840778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.840620    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.840843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.840816    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9ljm\" (UniqueName: \"kubernetes.io/projected/d0f3f304-7f78-425c-861a-e725febcc82c-kube-api-access-f9ljm\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.941666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.941601    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.941828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.941703    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.941866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.941833    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9ljm\" (UniqueName: \"kubernetes.io/projected/d0f3f304-7f78-425c-861a-e725febcc82c-kube-api-access-f9ljm\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.942364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.942344    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.944069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.944048    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:43.960213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:43.960183    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9ljm\" (UniqueName: \"kubernetes.io/projected/d0f3f304-7f78-425c-861a-e725febcc82c-kube-api-access-f9ljm\") pod \"maas-api-revoke-sub-1f64bfe5-966l8\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:44.105828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:44.105792    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:44.409879 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:28:44.409837    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0f3f304_7f78_425c_861a_e725febcc82c.slice/crio-712f1b60fe4caadefb8b62b68005c08a4dce589ada27767bfc05fdcde6953a38 WatchSource:0}: Error finding container 712f1b60fe4caadefb8b62b68005c08a4dce589ada27767bfc05fdcde6953a38: Status 404 returned error can't find the container with id 712f1b60fe4caadefb8b62b68005c08a4dce589ada27767bfc05fdcde6953a38
Oct 30 13:28:44.410419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:44.410331    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"]
Oct 30 13:28:45.359286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:45.359256    4291 generic.go:356] "Generic (PLEG): container finished" podID="d0f3f304-7f78-425c-861a-e725febcc82c" containerID="56abbcb8cc6fad5df183305e101e0483a8ceedfdaefcccfb63c99a8fe62dd21a" exitCode=0
Oct 30 13:28:45.359701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:45.359348    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8" event={"ID":"d0f3f304-7f78-425c-861a-e725febcc82c","Type":"ContainerDied","Data":"56abbcb8cc6fad5df183305e101e0483a8ceedfdaefcccfb63c99a8fe62dd21a"}
Oct 30 13:28:45.359701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:45.359408    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8" event={"ID":"d0f3f304-7f78-425c-861a-e725febcc82c","Type":"ContainerStarted","Data":"712f1b60fe4caadefb8b62b68005c08a4dce589ada27767bfc05fdcde6953a38"}
Oct 30 13:28:46.529826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.529796    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:46.574946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.573513    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0f3f304-7f78-425c-861a-e725febcc82c-kube-api-access-f9ljm\" (UniqueName: \"kubernetes.io/projected/d0f3f304-7f78-425c-861a-e725febcc82c-kube-api-access-f9ljm\") pod \"d0f3f304-7f78-425c-861a-e725febcc82c\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") "
Oct 30 13:28:46.574946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.573582    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-openshift-ca\") pod \"d0f3f304-7f78-425c-861a-e725febcc82c\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") "
Oct 30 13:28:46.574946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.573721    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-xks-ca\") pod \"d0f3f304-7f78-425c-861a-e725febcc82c\" (UID: \"d0f3f304-7f78-425c-861a-e725febcc82c\") "
Oct 30 13:28:46.574946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.574731    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-openshift-ca" pod "d0f3f304-7f78-425c-861a-e725febcc82c" (UID: "d0f3f304-7f78-425c-861a-e725febcc82c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:46.576881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.576848    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0f3f304-7f78-425c-861a-e725febcc82c-kube-api-access-f9ljm" pod "d0f3f304-7f78-425c-861a-e725febcc82c" (UID: "d0f3f304-7f78-425c-861a-e725febcc82c"). InnerVolumeSpecName "kube-api-access-f9ljm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:46.577010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.576928    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-xks-ca" pod "d0f3f304-7f78-425c-861a-e725febcc82c" (UID: "d0f3f304-7f78-425c-861a-e725febcc82c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:46.675707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.675595    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:46.675707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.675625    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9ljm\" (UniqueName: \"kubernetes.io/projected/d0f3f304-7f78-425c-861a-e725febcc82c-kube-api-access-f9ljm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:46.675707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:46.675671    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0f3f304-7f78-425c-861a-e725febcc82c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:47.374155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:47.374121    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"
Oct 30 13:28:47.374356 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:47.374152    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8" event={"ID":"d0f3f304-7f78-425c-861a-e725febcc82c","Type":"ContainerDied","Data":"712f1b60fe4caadefb8b62b68005c08a4dce589ada27767bfc05fdcde6953a38"}
Oct 30 13:28:47.374356 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:47.374250    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="712f1b60fe4caadefb8b62b68005c08a4dce589ada27767bfc05fdcde6953a38"
Oct 30 13:28:50.282974 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.282940    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"]
Oct 30 13:28:50.283542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.283525    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d0f3f304-7f78-425c-861a-e725febcc82c" containerName="revoke-keys"
Oct 30 13:28:50.283593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.283544    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0f3f304-7f78-425c-861a-e725febcc82c" containerName="revoke-keys"
Oct 30 13:28:50.283631 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.283626    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0f3f304-7f78-425c-861a-e725febcc82c" containerName="revoke-keys"
Oct 30 13:28:50.287514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.287484    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.303713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.303684    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"]
Oct 30 13:28:50.310075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.310044    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:50.310273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.310165    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:50.316362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.313971    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:28:50.316362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.314130    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:50.325585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.325547    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5wfkw\" (UniqueName: \"kubernetes.io/projected/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-kube-api-access-5wfkw\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.327861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.325853    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.327861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.325898    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.426598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.426562    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5wfkw\" (UniqueName: \"kubernetes.io/projected/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-kube-api-access-5wfkw\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.426843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.426704    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.426843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.426727    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.427353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.427327    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.429160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.429139    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.444956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.444928    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5wfkw\" (UniqueName: \"kubernetes.io/projected/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-kube-api-access-5wfkw\") pod \"maas-api-revoke-sub-c504d892-8rrt9\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.608118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.608080    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:50.923104 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:50.923078    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"]
Oct 30 13:28:50.924616 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:28:50.924589    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f02c58_ea03_4c86_8a61_50b8bbe2e962.slice/crio-6c855d6f9a7c0f4100e208c4b4effdb0a25b38bf3b0cf774d72341e5085a828a WatchSource:0}: Error finding container 6c855d6f9a7c0f4100e208c4b4effdb0a25b38bf3b0cf774d72341e5085a828a: Status 404 returned error can't find the container with id 6c855d6f9a7c0f4100e208c4b4effdb0a25b38bf3b0cf774d72341e5085a828a
Oct 30 13:28:51.402871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:51.402838    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9" event={"ID":"c1f02c58-ea03-4c86-8a61-50b8bbe2e962","Type":"ContainerStarted","Data":"bc9c7847db49d92a29e1237bb0165262b55c03847c414efaa75f784ee458825d"}
Oct 30 13:28:51.403311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:51.402890    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9" event={"ID":"c1f02c58-ea03-4c86-8a61-50b8bbe2e962","Type":"ContainerStarted","Data":"6c855d6f9a7c0f4100e208c4b4effdb0a25b38bf3b0cf774d72341e5085a828a"}
Oct 30 13:28:51.442341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:51.442290    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9" podStartSLOduration=1.442276858 podStartE2EDuration="1.442276858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:51.439097669 +0000 UTC m=+1541.699335909" watchObservedRunningTime="2026-10-30 13:28:51.442276858 +0000 UTC m=+1541.702515091"
Oct 30 13:28:52.411622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:52.411584    4291 generic.go:356] "Generic (PLEG): container finished" podID="c1f02c58-ea03-4c86-8a61-50b8bbe2e962" containerID="bc9c7847db49d92a29e1237bb0165262b55c03847c414efaa75f784ee458825d" exitCode=0
Oct 30 13:28:52.412090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:52.411676    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9" event={"ID":"c1f02c58-ea03-4c86-8a61-50b8bbe2e962","Type":"ContainerDied","Data":"bc9c7847db49d92a29e1237bb0165262b55c03847c414efaa75f784ee458825d"}
Oct 30 13:28:53.586952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.586924    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:53.671485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.671402    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-xks-ca\") pod \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") "
Oct 30 13:28:53.671485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.671481    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-kube-api-access-5wfkw\" (UniqueName: \"kubernetes.io/projected/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-kube-api-access-5wfkw\") pod \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") "
Oct 30 13:28:53.671765 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.671566    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-openshift-ca\") pod \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\" (UID: \"c1f02c58-ea03-4c86-8a61-50b8bbe2e962\") "
Oct 30 13:28:53.671935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.671907    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-openshift-ca" pod "c1f02c58-ea03-4c86-8a61-50b8bbe2e962" (UID: "c1f02c58-ea03-4c86-8a61-50b8bbe2e962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:28:53.672048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.672005    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:53.673624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.673598    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-xks-ca" pod "c1f02c58-ea03-4c86-8a61-50b8bbe2e962" (UID: "c1f02c58-ea03-4c86-8a61-50b8bbe2e962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:28:53.673718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.673651    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-kube-api-access-5wfkw" pod "c1f02c58-ea03-4c86-8a61-50b8bbe2e962" (UID: "c1f02c58-ea03-4c86-8a61-50b8bbe2e962"). InnerVolumeSpecName "kube-api-access-5wfkw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:28:53.772949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.772914    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:53.772949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:53.772942    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wfkw\" (UniqueName: \"kubernetes.io/projected/c1f02c58-ea03-4c86-8a61-50b8bbe2e962-kube-api-access-5wfkw\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:28:54.428271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:54.428240    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"
Oct 30 13:28:54.428449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:54.428271    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9" event={"ID":"c1f02c58-ea03-4c86-8a61-50b8bbe2e962","Type":"ContainerDied","Data":"6c855d6f9a7c0f4100e208c4b4effdb0a25b38bf3b0cf774d72341e5085a828a"}
Oct 30 13:28:54.428449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:54.428321    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c855d6f9a7c0f4100e208c4b4effdb0a25b38bf3b0cf774d72341e5085a828a"
Oct 30 13:28:56.700749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:56.700713    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:28:57.002292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.002200    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"]
Oct 30 13:28:57.002840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.002820    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c1f02c58-ea03-4c86-8a61-50b8bbe2e962" containerName="revoke-keys"
Oct 30 13:28:57.002957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.002841    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1f02c58-ea03-4c86-8a61-50b8bbe2e962" containerName="revoke-keys"
Oct 30 13:28:57.003031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.003018    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1f02c58-ea03-4c86-8a61-50b8bbe2e962" containerName="revoke-keys"
Oct 30 13:28:57.006681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.006659    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.022821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.022785    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"]
Oct 30 13:28:57.029363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.029332    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:28:57.029536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.029378    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:28:57.029536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.029513    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:28:57.029672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.029579    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:28:57.115358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.115321    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.115553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.115454    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24lv\" (UniqueName: \"kubernetes.io/projected/d3226023-6848-4f5d-bde3-933e0ff935b7-kube-api-access-z24lv\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.115553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.115516    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.216448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.216413    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.216678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.216597    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.216762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.216720    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24lv\" (UniqueName: \"kubernetes.io/projected/d3226023-6848-4f5d-bde3-933e0ff935b7-kube-api-access-z24lv\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.217165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.217142    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.218948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.218924    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.235505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.235477    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24lv\" (UniqueName: \"kubernetes.io/projected/d3226023-6848-4f5d-bde3-933e0ff935b7-kube-api-access-z24lv\") pod \"maas-api-revoke-sub-fda6aa59-xncwx\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.325503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.325415    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:28:57.809568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:57.807247    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"]
Oct 30 13:28:58.507450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:58.507416    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx" event={"ID":"d3226023-6848-4f5d-bde3-933e0ff935b7","Type":"ContainerStarted","Data":"c21e08a1800f8ce80bec75a2a58edb8d250eb35905cd1f4e298398208a497f3c"}
Oct 30 13:28:58.507777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:58.507752    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx" event={"ID":"d3226023-6848-4f5d-bde3-933e0ff935b7","Type":"ContainerStarted","Data":"e44211fa275afbeedf1b891d8a9da2fd3179ee87e9b24f2935a36a8b8d1e1e17"}
Oct 30 13:28:58.585663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:58.581750    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx" podStartSLOduration=2.581709889 podStartE2EDuration="2.581709889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:28:58.551070571 +0000 UTC m=+1548.811308805" watchObservedRunningTime="2026-10-30 13:28:58.581709889 +0000 UTC m=+1548.841948135"
Oct 30 13:28:59.609746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:59.609704    4291 generic.go:356] "Generic (PLEG): container finished" podID="d3226023-6848-4f5d-bde3-933e0ff935b7" containerID="c21e08a1800f8ce80bec75a2a58edb8d250eb35905cd1f4e298398208a497f3c" exitCode=0
Oct 30 13:28:59.615212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:28:59.610618    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx" event={"ID":"d3226023-6848-4f5d-bde3-933e0ff935b7","Type":"ContainerDied","Data":"c21e08a1800f8ce80bec75a2a58edb8d250eb35905cd1f4e298398208a497f3c"}
Oct 30 13:29:00.772363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.772336    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:29:00.915995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.915901    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3226023-6848-4f5d-bde3-933e0ff935b7-kube-api-access-z24lv\" (UniqueName: \"kubernetes.io/projected/d3226023-6848-4f5d-bde3-933e0ff935b7-kube-api-access-z24lv\") pod \"d3226023-6848-4f5d-bde3-933e0ff935b7\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") "
Oct 30 13:29:00.916208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.916000    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-openshift-ca\") pod \"d3226023-6848-4f5d-bde3-933e0ff935b7\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") "
Oct 30 13:29:00.916208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.916118    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-xks-ca\") pod \"d3226023-6848-4f5d-bde3-933e0ff935b7\" (UID: \"d3226023-6848-4f5d-bde3-933e0ff935b7\") "
Oct 30 13:29:00.916403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.916373    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-openshift-ca" pod "d3226023-6848-4f5d-bde3-933e0ff935b7" (UID: "d3226023-6848-4f5d-bde3-933e0ff935b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:00.916597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.916578    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:00.918018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.917994    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-xks-ca" pod "d3226023-6848-4f5d-bde3-933e0ff935b7" (UID: "d3226023-6848-4f5d-bde3-933e0ff935b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:00.918145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:00.918124    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3226023-6848-4f5d-bde3-933e0ff935b7-kube-api-access-z24lv" pod "d3226023-6848-4f5d-bde3-933e0ff935b7" (UID: "d3226023-6848-4f5d-bde3-933e0ff935b7"). InnerVolumeSpecName "kube-api-access-z24lv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:01.017972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:01.017937    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24lv\" (UniqueName: \"kubernetes.io/projected/d3226023-6848-4f5d-bde3-933e0ff935b7-kube-api-access-z24lv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:01.017972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:01.017969    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3226023-6848-4f5d-bde3-933e0ff935b7-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:01.627589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:01.627557    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"
Oct 30 13:29:01.627835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:01.627558    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx" event={"ID":"d3226023-6848-4f5d-bde3-933e0ff935b7","Type":"ContainerDied","Data":"e44211fa275afbeedf1b891d8a9da2fd3179ee87e9b24f2935a36a8b8d1e1e17"}
Oct 30 13:29:01.627835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:01.627780    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e44211fa275afbeedf1b891d8a9da2fd3179ee87e9b24f2935a36a8b8d1e1e17"
Oct 30 13:29:05.248756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.248723    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"]
Oct 30 13:29:05.249245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.249232    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d3226023-6848-4f5d-bde3-933e0ff935b7" containerName="revoke-keys"
Oct 30 13:29:05.249304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.249246    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3226023-6848-4f5d-bde3-933e0ff935b7" containerName="revoke-keys"
Oct 30 13:29:05.249352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.249344    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3226023-6848-4f5d-bde3-933e0ff935b7" containerName="revoke-keys"
Oct 30 13:29:05.253009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.252988    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.274331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.274295    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"]
Oct 30 13:29:05.279397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.279367    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:29:05.279564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.279473    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:29:05.279564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.279474    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:29:05.279564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.279527    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:29:05.338982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.338944    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.339249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.339222    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.339364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.339347    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z2hk\" (UniqueName: \"kubernetes.io/projected/dee515e5-6ec0-499f-866b-86bdadeeb41f-kube-api-access-8z2hk\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.440925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.440886    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z2hk\" (UniqueName: \"kubernetes.io/projected/dee515e5-6ec0-499f-866b-86bdadeeb41f-kube-api-access-8z2hk\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.441103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.441076    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.441153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.441121    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.441823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.441798    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.443761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.443735    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.460644 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.460596    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z2hk\" (UniqueName: \"kubernetes.io/projected/dee515e5-6ec0-499f-866b-86bdadeeb41f-kube-api-access-8z2hk\") pod \"maas-api-revoke-sub-00b53492-8c2sm\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.567314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.567238    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:05.876390 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:29:05.876357    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee515e5_6ec0_499f_866b_86bdadeeb41f.slice/crio-761dea91fb0b8bc1462dbae8916c85d287931ac7f6e828513bb80aa73314b885 WatchSource:0}: Error finding container 761dea91fb0b8bc1462dbae8916c85d287931ac7f6e828513bb80aa73314b885: Status 404 returned error can't find the container with id 761dea91fb0b8bc1462dbae8916c85d287931ac7f6e828513bb80aa73314b885
Oct 30 13:29:05.881749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:05.881723    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"]
Oct 30 13:29:06.673701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:06.673662    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm" event={"ID":"dee515e5-6ec0-499f-866b-86bdadeeb41f","Type":"ContainerStarted","Data":"8e15e14f7131d6d48c5cf0e42d62dfda8eea4f0bd2337753bf99674f5280275b"}
Oct 30 13:29:06.674191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:06.673723    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm" event={"ID":"dee515e5-6ec0-499f-866b-86bdadeeb41f","Type":"ContainerStarted","Data":"761dea91fb0b8bc1462dbae8916c85d287931ac7f6e828513bb80aa73314b885"}
Oct 30 13:29:06.711994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:06.711939    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm" podStartSLOduration=1.711925236 podStartE2EDuration="1.711925236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:06.710542465 +0000 UTC m=+1556.970780696" watchObservedRunningTime="2026-10-30 13:29:06.711925236 +0000 UTC m=+1556.972163469"
Oct 30 13:29:07.381621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.381575    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"]
Oct 30 13:29:07.385709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.385693    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.402897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.402865    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"]
Oct 30 13:29:07.469287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.469249    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.469486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.469308    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.469565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.469497    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdngk\" (UniqueName: \"kubernetes.io/projected/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-kube-api-access-rdngk\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.571210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.571170    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdngk\" (UniqueName: \"kubernetes.io/projected/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-kube-api-access-rdngk\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.571403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.571357    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.571477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.571434    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.572291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.572261    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.573822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.573793    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.589782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.589753    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdngk\" (UniqueName: \"kubernetes.io/projected/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-kube-api-access-rdngk\") pod \"maas-api-revoke-sub-81c628e3-8scgn\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:07.682326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.682241    4291 generic.go:356] "Generic (PLEG): container finished" podID="dee515e5-6ec0-499f-866b-86bdadeeb41f" containerID="8e15e14f7131d6d48c5cf0e42d62dfda8eea4f0bd2337753bf99674f5280275b" exitCode=0
Oct 30 13:29:07.682723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.682317    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm" event={"ID":"dee515e5-6ec0-499f-866b-86bdadeeb41f","Type":"ContainerDied","Data":"8e15e14f7131d6d48c5cf0e42d62dfda8eea4f0bd2337753bf99674f5280275b"}
Oct 30 13:29:07.702258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:07.702233    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:08.010118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.010027    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"]
Oct 30 13:29:08.014874 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:29:08.014843    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4895a2f_5c45_44a6_a54f_e7741ce13bc4.slice/crio-07b76ab6e74c4955eac02bae2f2494b80d2cc5037df104c202f842216bf1d99f WatchSource:0}: Error finding container 07b76ab6e74c4955eac02bae2f2494b80d2cc5037df104c202f842216bf1d99f: Status 404 returned error can't find the container with id 07b76ab6e74c4955eac02bae2f2494b80d2cc5037df104c202f842216bf1d99f
Oct 30 13:29:08.015356 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.015334    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"]
Oct 30 13:29:08.015458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.015433    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.032352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.032297    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"]
Oct 30 13:29:08.077524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.077486    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.077698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.077537    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.077758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.077735    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfst2\" (UniqueName: \"kubernetes.io/projected/7a7e0137-74aa-4b5d-9970-0afd68a39a02-kube-api-access-pfst2\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.178631 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.178593    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.178631 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.178648    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.178879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.178832    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfst2\" (UniqueName: \"kubernetes.io/projected/7a7e0137-74aa-4b5d-9970-0afd68a39a02-kube-api-access-pfst2\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.179314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.179292    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.180952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.180927    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.203379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.203345    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfst2\" (UniqueName: \"kubernetes.io/projected/7a7e0137-74aa-4b5d-9970-0afd68a39a02-kube-api-access-pfst2\") pod \"maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.215627 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:29:08.215590    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:29:08.377755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.377730    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:08.394739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.394711    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"]
Oct 30 13:29:08.394984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.394959    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy" containerID="cri-o://c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea" gracePeriod=30
Oct 30 13:29:08.398946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.398924    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"]
Oct 30 13:29:08.399150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.399124    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy" containerID="cri-o://1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745" gracePeriod=30
Oct 30 13:29:08.692943 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:29:08.692917    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a7e0137_74aa_4b5d_9970_0afd68a39a02.slice/crio-150fd44f6df5e21439ca685176334004be4c6f862f3c922fe26d72e0f8b9fb78 WatchSource:0}: Error finding container 150fd44f6df5e21439ca685176334004be4c6f862f3c922fe26d72e0f8b9fb78: Status 404 returned error can't find the container with id 150fd44f6df5e21439ca685176334004be4c6f862f3c922fe26d72e0f8b9fb78
Oct 30 13:29:08.694516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.694490    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"]
Oct 30 13:29:08.697937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.697899    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn" event={"ID":"a4895a2f-5c45-44a6-a54f-e7741ce13bc4","Type":"ContainerStarted","Data":"1ee0b072e2595479d05e019e617bda73405d71e4a5f09627f91660e1ab82f0d1"}
Oct 30 13:29:08.698083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.697986    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn" event={"ID":"a4895a2f-5c45-44a6-a54f-e7741ce13bc4","Type":"ContainerStarted","Data":"07b76ab6e74c4955eac02bae2f2494b80d2cc5037df104c202f842216bf1d99f"}
Oct 30 13:29:08.742982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.741812    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn" podStartSLOduration=1.7417923339999999 podStartE2EDuration="1.741792334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:08.737924415 +0000 UTC m=+1558.998162677" watchObservedRunningTime="2026-10-30 13:29:08.741792334 +0000 UTC m=+1559.002030568"
Oct 30 13:29:08.890333 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.890304    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:08.991306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.991271    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dee515e5-6ec0-499f-866b-86bdadeeb41f-kube-api-access-8z2hk\" (UniqueName: \"kubernetes.io/projected/dee515e5-6ec0-499f-866b-86bdadeeb41f-kube-api-access-8z2hk\") pod \"dee515e5-6ec0-499f-866b-86bdadeeb41f\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") "
Oct 30 13:29:08.991462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.991334    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-openshift-ca\") pod \"dee515e5-6ec0-499f-866b-86bdadeeb41f\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") "
Oct 30 13:29:08.991560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.991543    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-xks-ca\") pod \"dee515e5-6ec0-499f-866b-86bdadeeb41f\" (UID: \"dee515e5-6ec0-499f-866b-86bdadeeb41f\") "
Oct 30 13:29:08.991747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.991718    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-openshift-ca" pod "dee515e5-6ec0-499f-866b-86bdadeeb41f" (UID: "dee515e5-6ec0-499f-866b-86bdadeeb41f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:08.992226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.992203    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:08.993498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.993473    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dee515e5-6ec0-499f-866b-86bdadeeb41f-kube-api-access-8z2hk" pod "dee515e5-6ec0-499f-866b-86bdadeeb41f" (UID: "dee515e5-6ec0-499f-866b-86bdadeeb41f"). InnerVolumeSpecName "kube-api-access-8z2hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:08.993498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:08.993491    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-xks-ca" pod "dee515e5-6ec0-499f-866b-86bdadeeb41f" (UID: "dee515e5-6ec0-499f-866b-86bdadeeb41f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:09.093095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.093050    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dee515e5-6ec0-499f-866b-86bdadeeb41f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:09.093095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.093084    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8z2hk\" (UniqueName: \"kubernetes.io/projected/dee515e5-6ec0-499f-866b-86bdadeeb41f-kube-api-access-8z2hk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:09.714232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.713577    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"
Oct 30 13:29:09.714232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.713591    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm" event={"ID":"dee515e5-6ec0-499f-866b-86bdadeeb41f","Type":"ContainerDied","Data":"761dea91fb0b8bc1462dbae8916c85d287931ac7f6e828513bb80aa73314b885"}
Oct 30 13:29:09.714232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.713682    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="761dea91fb0b8bc1462dbae8916c85d287931ac7f6e828513bb80aa73314b885"
Oct 30 13:29:09.717714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.715995    4291 generic.go:356] "Generic (PLEG): container finished" podID="7a7e0137-74aa-4b5d-9970-0afd68a39a02" containerID="b7a4e73b9bbc4032b311f66422179cd020f099eb14590f72dc1154ddda09f703" exitCode=0
Oct 30 13:29:09.717714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.716213    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb" event={"ID":"7a7e0137-74aa-4b5d-9970-0afd68a39a02","Type":"ContainerDied","Data":"b7a4e73b9bbc4032b311f66422179cd020f099eb14590f72dc1154ddda09f703"}
Oct 30 13:29:09.717714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.716260    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb" event={"ID":"7a7e0137-74aa-4b5d-9970-0afd68a39a02","Type":"ContainerStarted","Data":"150fd44f6df5e21439ca685176334004be4c6f862f3c922fe26d72e0f8b9fb78"}
Oct 30 13:29:09.721094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.721064    4291 generic.go:356] "Generic (PLEG): container finished" podID="a4895a2f-5c45-44a6-a54f-e7741ce13bc4" containerID="1ee0b072e2595479d05e019e617bda73405d71e4a5f09627f91660e1ab82f0d1" exitCode=0
Oct 30 13:29:09.721228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:09.721127    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn" event={"ID":"a4895a2f-5c45-44a6-a54f-e7741ce13bc4","Type":"ContainerDied","Data":"1ee0b072e2595479d05e019e617bda73405d71e4a5f09627f91660e1ab82f0d1"}
Oct 30 13:29:10.928842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:10.928816    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:10.932218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:10.932197    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:11.026615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.025154    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-kube-api-access-rdngk\" (UniqueName: \"kubernetes.io/projected/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-kube-api-access-rdngk\") pod \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") "
Oct 30 13:29:11.026615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.025238    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-openshift-ca\") pod \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") "
Oct 30 13:29:11.026615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.025389    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a7e0137-74aa-4b5d-9970-0afd68a39a02-kube-api-access-pfst2\" (UniqueName: \"kubernetes.io/projected/7a7e0137-74aa-4b5d-9970-0afd68a39a02-kube-api-access-pfst2\") pod \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") "
Oct 30 13:29:11.026615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.025564    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-xks-ca\") pod \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\" (UID: \"7a7e0137-74aa-4b5d-9970-0afd68a39a02\") "
Oct 30 13:29:11.026615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.025681    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-openshift-ca\") pod \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") "
Oct 30 13:29:11.026615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.025899    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-xks-ca\") pod \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\" (UID: \"a4895a2f-5c45-44a6-a54f-e7741ce13bc4\") "
Oct 30 13:29:11.028341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.028309    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-openshift-ca" pod "a4895a2f-5c45-44a6-a54f-e7741ce13bc4" (UID: "a4895a2f-5c45-44a6-a54f-e7741ce13bc4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:11.028474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.028358    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-openshift-ca" pod "7a7e0137-74aa-4b5d-9970-0afd68a39a02" (UID: "7a7e0137-74aa-4b5d-9970-0afd68a39a02"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:11.033537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.032717    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-xks-ca" pod "7a7e0137-74aa-4b5d-9970-0afd68a39a02" (UID: "7a7e0137-74aa-4b5d-9970-0afd68a39a02"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:11.033537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.032872    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-kube-api-access-rdngk" pod "a4895a2f-5c45-44a6-a54f-e7741ce13bc4" (UID: "a4895a2f-5c45-44a6-a54f-e7741ce13bc4"). InnerVolumeSpecName "kube-api-access-rdngk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:11.033537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.033102    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a7e0137-74aa-4b5d-9970-0afd68a39a02-kube-api-access-pfst2" pod "7a7e0137-74aa-4b5d-9970-0afd68a39a02" (UID: "7a7e0137-74aa-4b5d-9970-0afd68a39a02"). InnerVolumeSpecName "kube-api-access-pfst2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:11.033537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.033180    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-xks-ca" pod "a4895a2f-5c45-44a6-a54f-e7741ce13bc4" (UID: "a4895a2f-5c45-44a6-a54f-e7741ce13bc4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:11.116505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.116470    4291 scope.go:122] "RemoveContainer" containerID="95ff3d3d02135e3f2e4db279cb0108e70b2392484c785add45f54895147ae5d1"
Oct 30 13:29:11.129125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.129097    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:11.129227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.129129    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:11.129227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.129144    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:11.129227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.129159    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdngk\" (UniqueName: \"kubernetes.io/projected/a4895a2f-5c45-44a6-a54f-e7741ce13bc4-kube-api-access-rdngk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:11.129227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.129174    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a7e0137-74aa-4b5d-9970-0afd68a39a02-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:11.129227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.129187    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfst2\" (UniqueName: \"kubernetes.io/projected/7a7e0137-74aa-4b5d-9970-0afd68a39a02-kube-api-access-pfst2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:11.741722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.741672    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb" event={"ID":"7a7e0137-74aa-4b5d-9970-0afd68a39a02","Type":"ContainerDied","Data":"150fd44f6df5e21439ca685176334004be4c6f862f3c922fe26d72e0f8b9fb78"}
Oct 30 13:29:11.741722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.741711    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"
Oct 30 13:29:11.741977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.741739    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="150fd44f6df5e21439ca685176334004be4c6f862f3c922fe26d72e0f8b9fb78"
Oct 30 13:29:11.743158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.743132    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn" event={"ID":"a4895a2f-5c45-44a6-a54f-e7741ce13bc4","Type":"ContainerDied","Data":"07b76ab6e74c4955eac02bae2f2494b80d2cc5037df104c202f842216bf1d99f"}
Oct 30 13:29:11.743158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.743152    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"
Oct 30 13:29:11.743372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:11.743172    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b76ab6e74c4955eac02bae2f2494b80d2cc5037df104c202f842216bf1d99f"
Oct 30 13:29:13.706884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.706857    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:29:13.709323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.709303    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:29:13.763767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.763734    4291 generic.go:356] "Generic (PLEG): container finished" podID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerID="c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea" exitCode=0
Oct 30 13:29:13.763962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.763788    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"
Oct 30 13:29:13.763962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.763817    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" event={"ID":"ce3603ce-72c0-4705-9aee-66df4fd80a25","Type":"ContainerDied","Data":"c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea"}
Oct 30 13:29:13.763962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.763876    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc" event={"ID":"ce3603ce-72c0-4705-9aee-66df4fd80a25","Type":"ContainerDied","Data":"b49065761f6037c6ff75496031de8a76b01398c99658608dda756e2e8861173d"}
Oct 30 13:29:13.763962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.763910    4291 scope.go:122] "RemoveContainer" containerID="c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea"
Oct 30 13:29:13.765825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.765804    4291 generic.go:356] "Generic (PLEG): container finished" podID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerID="1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745" exitCode=0
Oct 30 13:29:13.765947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.765833    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" event={"ID":"452f6b54-cb09-463e-acaa-0b4d046ac9a1","Type":"ContainerDied","Data":"1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745"}
Oct 30 13:29:13.765947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.765853    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"
Oct 30 13:29:13.765947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.765887    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r" event={"ID":"452f6b54-cb09-463e-acaa-0b4d046ac9a1","Type":"ContainerDied","Data":"dfb8ff54d85f69c88216514ee97249e6badaf6a8f8c4520bf477ecbe672fb22c"}
Oct 30 13:29:13.775458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.775434    4291 scope.go:122] "RemoveContainer" containerID="1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913"
Oct 30 13:29:13.785812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.785779    4291 scope.go:122] "RemoveContainer" containerID="c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea"
Oct 30 13:29:13.786143 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:29:13.786113    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea\": container with ID starting with c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea not found: ID does not exist" containerID="c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea"
Oct 30 13:29:13.786215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.786159    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea"} err="failed to get container status \"c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea\": rpc error: code = NotFound desc = could not find container \"c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea\": container with ID starting with c545777d41fd0d2f5beab9266f0054cb159663ddebf18ca626103e122a7c93ea not found: ID does not exist"
Oct 30 13:29:13.786215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.786185    4291 scope.go:122] "RemoveContainer" containerID="1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913"
Oct 30 13:29:13.786478 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:29:13.786457    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913\": container with ID starting with 1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913 not found: ID does not exist" containerID="1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913"
Oct 30 13:29:13.786521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.786488    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913"} err="failed to get container status \"1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913\": rpc error: code = NotFound desc = could not find container \"1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913\": container with ID starting with 1ad2b53fc79ab0fac341ea18033decf7abf9040664566e84c7d19a050767e913 not found: ID does not exist"
Oct 30 13:29:13.786521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.786505    4291 scope.go:122] "RemoveContainer" containerID="1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745"
Oct 30 13:29:13.798216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.798194    4291 scope.go:122] "RemoveContainer" containerID="1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745"
Oct 30 13:29:13.798795 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:29:13.798762    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745\": container with ID starting with 1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745 not found: ID does not exist" containerID="1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745"
Oct 30 13:29:13.798920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.798808    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745"} err="failed to get container status \"1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745\": rpc error: code = NotFound desc = could not find container \"1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745\": container with ID starting with 1cdeeca60868dba4fafbacb1d4302405b718870ffed8a343c03663fd9705c745 not found: ID does not exist"
Oct 30 13:29:13.862753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.862715    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-podinfo\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.862958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.862801    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-certs\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.862958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.862842    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-credential-socket\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.862958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.862872    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-token\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-token\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.862958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.862900    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-podinfo\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.862958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.862926    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-kube-api-access-b5wnd\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-kube-api-access-b5wnd\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.863224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.862968    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-envoy\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.863224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863016    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istiod-ca-cert\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.863224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863078    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-kube-api-access-9b42n\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-kube-api-access-9b42n\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.863224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863119    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-certs\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.863224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863137    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-certs" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.863224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863156    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-credential-socket" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.863224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863159    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-credential-socket\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.863595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863249    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-data\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.863595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863280    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-data\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.863595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863303    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-envoy\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.863595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863382    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-token\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-token\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.863595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863459    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce3603ce-72c0-4705-9aee-66df4fd80a25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3603ce-72c0-4705-9aee-66df4fd80a25-istiod-ca-cert\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.863595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863571    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-socket\") pod \"ce3603ce-72c0-4705-9aee-66df4fd80a25\" (UID: \"ce3603ce-72c0-4705-9aee-66df4fd80a25\") "
Oct 30 13:29:13.863595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863572    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-certs" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.864024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863698    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-socket\") pod \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\" (UID: \"452f6b54-cb09-463e-acaa-0b4d046ac9a1\") "
Oct 30 13:29:13.864024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.863766    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-credential-socket" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.864186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.864158    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istiod-ca-cert" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:13.864309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.864237    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce3603ce-72c0-4705-9aee-66df4fd80a25-istiod-ca-cert" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:13.864414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.864391    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-socket" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.864586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.864562    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-data" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.864747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.864547    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-data" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.864851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.864606    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-socket" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.865081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865062    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865170    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865283 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865262    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865286    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865302    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865317    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865329    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865342    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865355    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.865384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.865370    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ce3603ce-72c0-4705-9aee-66df4fd80a25-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.866243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.866217    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-envoy" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.866419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.866395    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-podinfo" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:29:13.866535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.866509    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-token" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:13.866836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.866811    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-kube-api-access-b5wnd" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "kube-api-access-b5wnd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:13.866915 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.866812    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-podinfo" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:29:13.866960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.866920    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-token" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:13.867352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.867334    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-kube-api-access-9b42n" pod "452f6b54-cb09-463e-acaa-0b4d046ac9a1" (UID: "452f6b54-cb09-463e-acaa-0b4d046ac9a1"). InnerVolumeSpecName "kube-api-access-9b42n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:13.867573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.867542    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-envoy" pod "ce3603ce-72c0-4705-9aee-66df4fd80a25" (UID: "ce3603ce-72c0-4705-9aee-66df4fd80a25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:13.967304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967207    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.967304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967241    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.967304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967253    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.967304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967264    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.967304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967275    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ce3603ce-72c0-4705-9aee-66df4fd80a25-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.967304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967286    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5wnd\" (UniqueName: \"kubernetes.io/projected/ce3603ce-72c0-4705-9aee-66df4fd80a25-kube-api-access-b5wnd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.967304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967300    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/452f6b54-cb09-463e-acaa-0b4d046ac9a1-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:13.967658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:13.967315    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b42n\" (UniqueName: \"kubernetes.io/projected/452f6b54-cb09-463e-acaa-0b4d046ac9a1-kube-api-access-9b42n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:14.106014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:14.105979    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"]
Oct 30 13:29:14.117868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:14.117833    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-jw59r"]
Oct 30 13:29:14.145018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:14.144974    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"]
Oct 30 13:29:14.148786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:14.148758    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-bhnqc"]
Oct 30 13:29:15.881835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:15.881800    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" path="/var/lib/kubelet/pods/452f6b54-cb09-463e-acaa-0b4d046ac9a1/volumes"
Oct 30 13:29:15.882459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:15.882443    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" path="/var/lib/kubelet/pods/ce3603ce-72c0-4705-9aee-66df4fd80a25/volumes"
Oct 30 13:29:18.345516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:18.345471    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"]
Oct 30 13:29:18.347016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:18.346959    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy" containerID="cri-o://c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669" gracePeriod=30
Oct 30 13:29:23.640773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.640715    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:29:23.735752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.735721    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-data\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.735941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.735818    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-podinfo\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.735941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.735860    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-credential-socket\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.735941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.735920    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-token\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-token\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.735941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.735942    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50f2b6e2-8cf4-437f-97f8-573f66633fec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f2b6e2-8cf4-437f-97f8-573f66633fec-istiod-ca-cert\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.736171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.735960    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-kube-api-access-4zx4p\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-kube-api-access-4zx4p\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.736171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736008    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-data" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:23.736171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736045    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-envoy\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.736171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736114    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-certs\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.736171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736154    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-socket\") pod \"50f2b6e2-8cf4-437f-97f8-573f66633fec\" (UID: \"50f2b6e2-8cf4-437f-97f8-573f66633fec\") "
Oct 30 13:29:23.736407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736229    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-credential-socket" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:23.736407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736333    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50f2b6e2-8cf4-437f-97f8-573f66633fec-istiod-ca-cert" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:23.736521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736501    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-certs" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:23.736602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736578    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-socket" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:23.736756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736735    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.736826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736762    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.736826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736772    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.736826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736789    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.736826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.736799    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/50f2b6e2-8cf4-437f-97f8-573f66633fec-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.738189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.738169    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-podinfo" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:29:23.738292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.738280    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-token" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:23.738577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.738557    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-envoy" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:29:23.738630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.738585    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-kube-api-access-4zx4p" pod "50f2b6e2-8cf4-437f-97f8-573f66633fec" (UID: "50f2b6e2-8cf4-437f-97f8-573f66633fec"). InnerVolumeSpecName "kube-api-access-4zx4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:23.818255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818225    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"]
Oct 30 13:29:23.818780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818762    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy"
Oct 30 13:29:23.818880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818780    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy"
Oct 30 13:29:23.818880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818799    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a4895a2f-5c45-44a6-a54f-e7741ce13bc4" containerName="revoke-keys"
Oct 30 13:29:23.818880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818807    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4895a2f-5c45-44a6-a54f-e7741ce13bc4" containerName="revoke-keys"
Oct 30 13:29:23.818880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818832    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy"
Oct 30 13:29:23.818880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818840    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy"
Oct 30 13:29:23.818880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818866    4291 cpu_manager.go:414] "containerMap: removing container" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy"
Oct 30 13:29:23.818880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818873    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818893    4291 cpu_manager.go:414] "containerMap: removing container" podUID="dee515e5-6ec0-499f-866b-86bdadeeb41f" containerName="revoke-keys"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818900    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dee515e5-6ec0-499f-866b-86bdadeeb41f" containerName="revoke-keys"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818910    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7a7e0137-74aa-4b5d-9970-0afd68a39a02" containerName="revoke-keys"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818917    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a7e0137-74aa-4b5d-9970-0afd68a39a02" containerName="revoke-keys"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818933    4291 cpu_manager.go:414] "containerMap: removing container" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818940    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818978    4291 cpu_manager.go:414] "containerMap: removing container" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.818985    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819106    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="452f6b54-cb09-463e-acaa-0b4d046ac9a1" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819121    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="dee515e5-6ec0-499f-866b-86bdadeeb41f" containerName="revoke-keys"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819130    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a7e0137-74aa-4b5d-9970-0afd68a39a02" containerName="revoke-keys"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819139    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819147    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819159    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819168    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4895a2f-5c45-44a6-a54f-e7741ce13bc4" containerName="revoke-keys"
Oct 30 13:29:23.819234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.819178    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce3603ce-72c0-4705-9aee-66df4fd80a25" containerName="istio-proxy"
Oct 30 13:29:23.822999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.822980    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:23.837441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837417    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:29:23.837441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837421    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:29:23.837624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837447    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:29:23.837624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837513    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:29:23.837753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837691    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.837753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837716    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zx4p\" (UniqueName: \"kubernetes.io/projected/50f2b6e2-8cf4-437f-97f8-573f66633fec-kube-api-access-4zx4p\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.837753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837729    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.837753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.837741    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/50f2b6e2-8cf4-437f-97f8-573f66633fec-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:23.839855 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.839835    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"]
Oct 30 13:29:23.897561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.897498    4291 generic.go:356] "Generic (PLEG): container finished" podID="50f2b6e2-8cf4-437f-97f8-573f66633fec" containerID="c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669" exitCode=0
Oct 30 13:29:23.897561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.897545    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"
Oct 30 13:29:23.897759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.897579    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" event={"ID":"50f2b6e2-8cf4-437f-97f8-573f66633fec","Type":"ContainerDied","Data":"c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669"}
Oct 30 13:29:23.897759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.897692    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs" event={"ID":"50f2b6e2-8cf4-437f-97f8-573f66633fec","Type":"ContainerDied","Data":"5192332f31f936143956921e472a713dc0339f9e43ab9718b5c73e30a97d17c2"}
Oct 30 13:29:23.897759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.897737    4291 scope.go:122] "RemoveContainer" containerID="c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669"
Oct 30 13:29:23.908944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.908926    4291 scope.go:122] "RemoveContainer" containerID="23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab"
Oct 30 13:29:23.923072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.923045    4291 scope.go:122] "RemoveContainer" containerID="c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669"
Oct 30 13:29:23.923360 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:29:23.923342    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669\": container with ID starting with c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669 not found: ID does not exist" containerID="c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669"
Oct 30 13:29:23.923440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.923367    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669"} err="failed to get container status \"c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669\": rpc error: code = NotFound desc = could not find container \"c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669\": container with ID starting with c6c615e0726a616384798e3391e78a4b23dc0c67c65d00a5cd0b0f6b41cb3669 not found: ID does not exist"
Oct 30 13:29:23.923440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.923384    4291 scope.go:122] "RemoveContainer" containerID="23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab"
Oct 30 13:29:23.923739 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:29:23.923715    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab\": container with ID starting with 23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab not found: ID does not exist" containerID="23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab"
Oct 30 13:29:23.923847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.923745    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab"} err="failed to get container status \"23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab\": rpc error: code = NotFound desc = could not find container \"23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab\": container with ID starting with 23dbd80355995ae869cbb4195a2a385025eed0e10f0f1b14ef9207b064a5eaab not found: ID does not exist"
Oct 30 13:29:23.938492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.938470    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:23.938606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.938518    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:23.938720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.938700    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcdjc\" (UniqueName: \"kubernetes.io/projected/a3c12e9f-505f-47b3-94ff-ffec605ae63a-kube-api-access-kcdjc\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:23.949881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.949827    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"]
Oct 30 13:29:23.955614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:23.955593    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5cee02-openshift-default-65877fc697-47wbs"]
Oct 30 13:29:24.040287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.040257    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcdjc\" (UniqueName: \"kubernetes.io/projected/a3c12e9f-505f-47b3-94ff-ffec605ae63a-kube-api-access-kcdjc\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:24.040407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.040360    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:24.040451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.040404    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:24.041117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.041097    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:24.042776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.042760    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:24.059000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.058980    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcdjc\" (UniqueName: \"kubernetes.io/projected/a3c12e9f-505f-47b3-94ff-ffec605ae63a-kube-api-access-kcdjc\") pod \"maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:24.137001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.136982    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:24.441107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.441078    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"]
Oct 30 13:29:24.442096 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:29:24.442069    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c12e9f_505f_47b3_94ff_ffec605ae63a.slice/crio-2cc667133b8ccd987724f13a514a7c2fe6caa94437e7e35972d8186d9508a0e1 WatchSource:0}: Error finding container 2cc667133b8ccd987724f13a514a7c2fe6caa94437e7e35972d8186d9508a0e1: Status 404 returned error can't find the container with id 2cc667133b8ccd987724f13a514a7c2fe6caa94437e7e35972d8186d9508a0e1
Oct 30 13:29:24.907054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.907015    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t" event={"ID":"a3c12e9f-505f-47b3-94ff-ffec605ae63a","Type":"ContainerStarted","Data":"cb405afc95009a42b6f7195b97780150bc081ea370efa73cd5a152b3f03b24cc"}
Oct 30 13:29:24.907446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.907075    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t" event={"ID":"a3c12e9f-505f-47b3-94ff-ffec605ae63a","Type":"ContainerStarted","Data":"2cc667133b8ccd987724f13a514a7c2fe6caa94437e7e35972d8186d9508a0e1"}
Oct 30 13:29:24.952648 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:24.952584    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t" podStartSLOduration=1.95257024 podStartE2EDuration="1.95257024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:24.948948857 +0000 UTC m=+1575.209187099" watchObservedRunningTime="2026-10-30 13:29:24.95257024 +0000 UTC m=+1575.212808475"
Oct 30 13:29:25.878046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:25.878001    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50f2b6e2-8cf4-437f-97f8-573f66633fec" path="/var/lib/kubelet/pods/50f2b6e2-8cf4-437f-97f8-573f66633fec/volumes"
Oct 30 13:29:25.915593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:25.915558    4291 generic.go:356] "Generic (PLEG): container finished" podID="a3c12e9f-505f-47b3-94ff-ffec605ae63a" containerID="cb405afc95009a42b6f7195b97780150bc081ea370efa73cd5a152b3f03b24cc" exitCode=0
Oct 30 13:29:25.916014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:25.915662    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t" event={"ID":"a3c12e9f-505f-47b3-94ff-ffec605ae63a","Type":"ContainerDied","Data":"cb405afc95009a42b6f7195b97780150bc081ea370efa73cd5a152b3f03b24cc"}
Oct 30 13:29:27.135217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.135194    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:27.182432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.180290    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3c12e9f-505f-47b3-94ff-ffec605ae63a-kube-api-access-kcdjc\" (UniqueName: \"kubernetes.io/projected/a3c12e9f-505f-47b3-94ff-ffec605ae63a-kube-api-access-kcdjc\") pod \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") "
Oct 30 13:29:27.182432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.180442    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-openshift-ca\") pod \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") "
Oct 30 13:29:27.182432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.180504    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-xks-ca\") pod \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\" (UID: \"a3c12e9f-505f-47b3-94ff-ffec605ae63a\") "
Oct 30 13:29:27.184113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.183711    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-xks-ca" pod "a3c12e9f-505f-47b3-94ff-ffec605ae63a" (UID: "a3c12e9f-505f-47b3-94ff-ffec605ae63a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:27.185059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.184970    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-openshift-ca" pod "a3c12e9f-505f-47b3-94ff-ffec605ae63a" (UID: "a3c12e9f-505f-47b3-94ff-ffec605ae63a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:27.208415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.208384    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3c12e9f-505f-47b3-94ff-ffec605ae63a-kube-api-access-kcdjc" pod "a3c12e9f-505f-47b3-94ff-ffec605ae63a" (UID: "a3c12e9f-505f-47b3-94ff-ffec605ae63a"). InnerVolumeSpecName "kube-api-access-kcdjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:27.286712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.286674    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcdjc\" (UniqueName: \"kubernetes.io/projected/a3c12e9f-505f-47b3-94ff-ffec605ae63a-kube-api-access-kcdjc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.286712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.286714    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.286994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.286729    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3c12e9f-505f-47b3-94ff-ffec605ae63a-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:27.706084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.706052    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"]
Oct 30 13:29:27.717958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.717935    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65e250-c9bkb"]
Oct 30 13:29:27.893560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.893525    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a7e0137-74aa-4b5d-9970-0afd68a39a02" path="/var/lib/kubelet/pods/7a7e0137-74aa-4b5d-9970-0afd68a39a02/volumes"
Oct 30 13:29:27.941083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.941051    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"
Oct 30 13:29:27.941289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.941053    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t" event={"ID":"a3c12e9f-505f-47b3-94ff-ffec605ae63a","Type":"ContainerDied","Data":"2cc667133b8ccd987724f13a514a7c2fe6caa94437e7e35972d8186d9508a0e1"}
Oct 30 13:29:27.941289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:27.941155    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cc667133b8ccd987724f13a514a7c2fe6caa94437e7e35972d8186d9508a0e1"
Oct 30 13:29:35.689664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.689620    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"]
Oct 30 13:29:35.690100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.690087    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a3c12e9f-505f-47b3-94ff-ffec605ae63a" containerName="revoke-keys"
Oct 30 13:29:35.690147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.690101    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3c12e9f-505f-47b3-94ff-ffec605ae63a" containerName="revoke-keys"
Oct 30 13:29:35.690200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.690191    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3c12e9f-505f-47b3-94ff-ffec605ae63a" containerName="revoke-keys"
Oct 30 13:29:35.693498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.693481    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.710493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.710467    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"]
Oct 30 13:29:35.718276 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.718242    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:29:35.718489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.718469    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:29:35.732333 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.732310    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:29:35.732612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.732594    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:29:35.788225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.788192    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.788405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.788232    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr8s5\" (UniqueName: \"kubernetes.io/projected/51d9c19b-bc05-49da-a2ca-481c52e9080b-kube-api-access-nr8s5\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.788405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.788358    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.889227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.889191    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.889227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.889233    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr8s5\" (UniqueName: \"kubernetes.io/projected/51d9c19b-bc05-49da-a2ca-481c52e9080b-kube-api-access-nr8s5\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.889482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.889300    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.889933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.889905    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.891715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.891697    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:35.907842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:35.907817    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr8s5\" (UniqueName: \"kubernetes.io/projected/51d9c19b-bc05-49da-a2ca-481c52e9080b-kube-api-access-nr8s5\") pod \"maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:36.006996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:36.006924    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:36.309426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:36.309396    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"]
Oct 30 13:29:36.310585 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:29:36.310554    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51d9c19b_bc05_49da_a2ca_481c52e9080b.slice/crio-89b1442a957613a2b084dd050b795de1f6560fd18065452c95c5d7ed6e52ee7c WatchSource:0}: Error finding container 89b1442a957613a2b084dd050b795de1f6560fd18065452c95c5d7ed6e52ee7c: Status 404 returned error can't find the container with id 89b1442a957613a2b084dd050b795de1f6560fd18065452c95c5d7ed6e52ee7c
Oct 30 13:29:37.005994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:37.005960    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc" event={"ID":"51d9c19b-bc05-49da-a2ca-481c52e9080b","Type":"ContainerStarted","Data":"13f99fa7c06ca3f9e9738497ab7d5bfc7a096eee4f0a87d2c9ce143100317bab"}
Oct 30 13:29:37.006439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:37.006014    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc" event={"ID":"51d9c19b-bc05-49da-a2ca-481c52e9080b","Type":"ContainerStarted","Data":"89b1442a957613a2b084dd050b795de1f6560fd18065452c95c5d7ed6e52ee7c"}
Oct 30 13:29:37.046568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:37.046521    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc" podStartSLOduration=2.046509649 podStartE2EDuration="2.046509649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:29:37.043309728 +0000 UTC m=+1587.303547960" watchObservedRunningTime="2026-10-30 13:29:37.046509649 +0000 UTC m=+1587.306747883"
Oct 30 13:29:38.014369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:38.014335    4291 generic.go:356] "Generic (PLEG): container finished" podID="51d9c19b-bc05-49da-a2ca-481c52e9080b" containerID="13f99fa7c06ca3f9e9738497ab7d5bfc7a096eee4f0a87d2c9ce143100317bab" exitCode=0
Oct 30 13:29:38.014782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:38.014421    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc" event={"ID":"51d9c19b-bc05-49da-a2ca-481c52e9080b","Type":"ContainerDied","Data":"13f99fa7c06ca3f9e9738497ab7d5bfc7a096eee4f0a87d2c9ce143100317bab"}
Oct 30 13:29:38.209491 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:29:38.209451    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:29:38.744753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:38.744721    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:29:39.172164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.172140    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:39.329773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.329678    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-openshift-ca\") pod \"51d9c19b-bc05-49da-a2ca-481c52e9080b\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") "
Oct 30 13:29:39.329923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.329809    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-xks-ca\") pod \"51d9c19b-bc05-49da-a2ca-481c52e9080b\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") "
Oct 30 13:29:39.329984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.329923    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d9c19b-bc05-49da-a2ca-481c52e9080b-kube-api-access-nr8s5\" (UniqueName: \"kubernetes.io/projected/51d9c19b-bc05-49da-a2ca-481c52e9080b-kube-api-access-nr8s5\") pod \"51d9c19b-bc05-49da-a2ca-481c52e9080b\" (UID: \"51d9c19b-bc05-49da-a2ca-481c52e9080b\") "
Oct 30 13:29:39.330053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.330018    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-openshift-ca" pod "51d9c19b-bc05-49da-a2ca-481c52e9080b" (UID: "51d9c19b-bc05-49da-a2ca-481c52e9080b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:29:39.330301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.330286    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:39.332015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.331992    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d9c19b-bc05-49da-a2ca-481c52e9080b-kube-api-access-nr8s5" pod "51d9c19b-bc05-49da-a2ca-481c52e9080b" (UID: "51d9c19b-bc05-49da-a2ca-481c52e9080b"). InnerVolumeSpecName "kube-api-access-nr8s5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:29:39.332108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.332069    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-xks-ca" pod "51d9c19b-bc05-49da-a2ca-481c52e9080b" (UID: "51d9c19b-bc05-49da-a2ca-481c52e9080b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:29:39.431340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.431305    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51d9c19b-bc05-49da-a2ca-481c52e9080b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:39.431340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:39.431342    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr8s5\" (UniqueName: \"kubernetes.io/projected/51d9c19b-bc05-49da-a2ca-481c52e9080b-kube-api-access-nr8s5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:29:40.030768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:40.030737    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"
Oct 30 13:29:40.030768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:40.030753    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc" event={"ID":"51d9c19b-bc05-49da-a2ca-481c52e9080b","Type":"ContainerDied","Data":"89b1442a957613a2b084dd050b795de1f6560fd18065452c95c5d7ed6e52ee7c"}
Oct 30 13:29:40.030963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:40.030803    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="89b1442a957613a2b084dd050b795de1f6560fd18065452c95c5d7ed6e52ee7c"
Oct 30 13:29:44.840838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:44.840805    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"]
Oct 30 13:29:44.852164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:44.852131    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-4091a5-scb6t"]
Oct 30 13:29:45.893417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:45.893362    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3c12e9f-505f-47b3-94ff-ffec605ae63a" path="/var/lib/kubelet/pods/a3c12e9f-505f-47b3-94ff-ffec605ae63a/volumes"
Oct 30 13:29:49.853097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:49.853059    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"]
Oct 30 13:29:49.865061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:49.865026    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-642765-6m2dc"]
Oct 30 13:29:49.887972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:29:49.887932    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d9c19b-bc05-49da-a2ca-481c52e9080b" path="/var/lib/kubelet/pods/51d9c19b-bc05-49da-a2ca-481c52e9080b/volumes"
Oct 30 13:30:00.150378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.150330    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"]
Oct 30 13:30:00.151385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.151359    4291 cpu_manager.go:414] "containerMap: removing container" podUID="51d9c19b-bc05-49da-a2ca-481c52e9080b" containerName="revoke-keys"
Oct 30 13:30:00.151385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.151385    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d9c19b-bc05-49da-a2ca-481c52e9080b" containerName="revoke-keys"
Oct 30 13:30:00.151564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.151521    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d9c19b-bc05-49da-a2ca-481c52e9080b" containerName="revoke-keys"
Oct 30 13:30:00.156260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.156233    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"]
Oct 30 13:30:00.156404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.156361    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.160371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.160340    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.161037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.160991    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"]
Oct 30 13:30:00.165603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.165577    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"]
Oct 30 13:30:00.165778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.165699    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.174749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.174713    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"]
Oct 30 13:30:00.179565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.179533    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"]
Oct 30 13:30:00.184009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.183985    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:30:00.184185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.184169    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:30:00.184281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.184263    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:30:00.189157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.189123    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q5n8\" (UniqueName: \"kubernetes.io/projected/672131ae-497d-48e1-93f7-c1497cdd8f2b-kube-api-access-7q5n8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.189290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.189202    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd9nn\" (UniqueName: \"kubernetes.io/projected/533b7741-937a-461f-8030-04da2f8af69c-kube-api-access-bd9nn\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.189290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.189240    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.189290 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.189262    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh5h7\" (UniqueName: \"kubernetes.io/projected/9f98cd42-d637-4f52-8e4c-54447615de8d-kube-api-access-dh5h7\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.189481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.189296    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f98cd42-d637-4f52-8e4c-54447615de8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.189481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.189402    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b7741-937a-461f-8030-04da2f8af69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.189603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.189568    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672131ae-497d-48e1-93f7-c1497cdd8f2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.190274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.190254    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"]
Oct 30 13:30:00.194985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.194960    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"]
Oct 30 13:30:00.199358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.199316    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.205617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.205589    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"]
Oct 30 13:30:00.217204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.217155    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"]
Oct 30 13:30:00.290866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.290831    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hnf8n\" (UniqueName: \"kubernetes.io/projected/c4a553d3-8e02-4ab2-b0f6-044873b091b7-kube-api-access-hnf8n\") pod \"maas-api-key-cleanup-29889450-wk4rz\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.290866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.290871    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e5dcb1-b9b2-4ed3-8115-96945314789e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.291126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.290973    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b7741-937a-461f-8030-04da2f8af69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.291126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291024    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672131ae-497d-48e1-93f7-c1497cdd8f2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.291126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291082    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a553d3-8e02-4ab2-b0f6-044873b091b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-wk4rz\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.291126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291105    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q5n8\" (UniqueName: \"kubernetes.io/projected/672131ae-497d-48e1-93f7-c1497cdd8f2b-kube-api-access-7q5n8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.291344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291177    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wfwx\" (UniqueName: \"kubernetes.io/projected/18e5dcb1-b9b2-4ed3-8115-96945314789e-kube-api-access-9wfwx\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.291344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291205    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd9nn\" (UniqueName: \"kubernetes.io/projected/533b7741-937a-461f-8030-04da2f8af69c-kube-api-access-bd9nn\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.291452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291345    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh5h7\" (UniqueName: \"kubernetes.io/projected/9f98cd42-d637-4f52-8e4c-54447615de8d-kube-api-access-dh5h7\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.291452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291399    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f98cd42-d637-4f52-8e4c-54447615de8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.291830 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291808    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b7741-937a-461f-8030-04da2f8af69c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.291930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.291809    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672131ae-497d-48e1-93f7-c1497cdd8f2b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.292064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.292046    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f98cd42-d637-4f52-8e4c-54447615de8d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.310308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.310271    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh5h7\" (UniqueName: \"kubernetes.io/projected/9f98cd42-d637-4f52-8e4c-54447615de8d-kube-api-access-dh5h7\") pod \"maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.310308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.310307    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd9nn\" (UniqueName: \"kubernetes.io/projected/533b7741-937a-461f-8030-04da2f8af69c-kube-api-access-bd9nn\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.310547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.310327    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q5n8\" (UniqueName: \"kubernetes.io/projected/672131ae-497d-48e1-93f7-c1497cdd8f2b-kube-api-access-7q5n8\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.392657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.392605    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a553d3-8e02-4ab2-b0f6-044873b091b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-wk4rz\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.392876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.392698    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wfwx\" (UniqueName: \"kubernetes.io/projected/18e5dcb1-b9b2-4ed3-8115-96945314789e-kube-api-access-9wfwx\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.392876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.392781    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hnf8n\" (UniqueName: \"kubernetes.io/projected/c4a553d3-8e02-4ab2-b0f6-044873b091b7-kube-api-access-hnf8n\") pod \"maas-api-key-cleanup-29889450-wk4rz\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.392876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.392805    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e5dcb1-b9b2-4ed3-8115-96945314789e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.393312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.393290    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a553d3-8e02-4ab2-b0f6-044873b091b7-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889450-wk4rz\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.393448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.393427    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e5dcb1-b9b2-4ed3-8115-96945314789e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.411303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.411223    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wfwx\" (UniqueName: \"kubernetes.io/projected/18e5dcb1-b9b2-4ed3-8115-96945314789e-kube-api-access-9wfwx\") pod \"maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.411303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.411269    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hnf8n\" (UniqueName: \"kubernetes.io/projected/c4a553d3-8e02-4ab2-b0f6-044873b091b7-kube-api-access-hnf8n\") pod \"maas-api-key-cleanup-29889450-wk4rz\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.488894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.488860    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:00.508055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.508028    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:00.523393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.523360    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:00.535308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.535271    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:00.542325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.542114    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:00.826185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:00.824703    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"]
Oct 30 13:30:00.826185 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:30:00.825246    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672131ae_497d_48e1_93f7_c1497cdd8f2b.slice/crio-28823a1e594dbfc6439f0dc888635329d0f944188f45eb5a5f6b596a29bbbbc8 WatchSource:0}: Error finding container 28823a1e594dbfc6439f0dc888635329d0f944188f45eb5a5f6b596a29bbbbc8: Status 404 returned error can't find the container with id 28823a1e594dbfc6439f0dc888635329d0f944188f45eb5a5f6b596a29bbbbc8
Oct 30 13:30:01.051367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.050491    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"]
Oct 30 13:30:01.055426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.055402    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"]
Oct 30 13:30:01.060850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.060519    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"]
Oct 30 13:30:01.089934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.089876    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"]
Oct 30 13:30:01.219097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.219061    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"]
Oct 30 13:30:01.223144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.223110    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" event={"ID":"9f98cd42-d637-4f52-8e4c-54447615de8d","Type":"ContainerStarted","Data":"781d9363996566dc3220c1ed534e24d77582a24cb309fd54953c8aba64efc756"}
Oct 30 13:30:01.223316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.223232    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.224738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.224705    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" event={"ID":"c4a553d3-8e02-4ab2-b0f6-044873b091b7","Type":"ContainerStarted","Data":"d266531d2b0ce3dedbbf7a23e601753c4e4b3a2ac3a2161735d32a58cee39218"}
Oct 30 13:30:01.226449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.226420    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerStarted","Data":"c8160ab8cf619ec0816ec141f7ea9f549e55edf52776ae27cb63038070edbc24"}
Oct 30 13:30:01.226574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.226472    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerStarted","Data":"28823a1e594dbfc6439f0dc888635329d0f944188f45eb5a5f6b596a29bbbbc8"}
Oct 30 13:30:01.227841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.227818    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" event={"ID":"18e5dcb1-b9b2-4ed3-8115-96945314789e","Type":"ContainerStarted","Data":"7e23f637494672fea552d785b7247ffef9221d91bd9be54cb0cc2d3aeff7c422"}
Oct 30 13:30:01.229259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.229241    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" event={"ID":"533b7741-937a-461f-8030-04da2f8af69c","Type":"ContainerStarted","Data":"dfc0417b732b1971fff5123bd795804439f0b3eadb8587144370f3877af5c158"}
Oct 30 13:30:01.237936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.237879    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:30:01.255691 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.246360    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"]
Oct 30 13:30:01.298796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.298657    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" podStartSLOduration=1.298622297 podStartE2EDuration="1.298622297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:01.295895059 +0000 UTC m=+1611.556133296" watchObservedRunningTime="2026-10-30 13:30:01.298622297 +0000 UTC m=+1611.558860531"
Oct 30 13:30:01.314581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.314538    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7prbx\" (UniqueName: \"kubernetes.io/projected/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-kube-api-access-7prbx\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.314809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.314790    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.314876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.314862    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.416077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.416035    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.416077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.416095    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.416303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.416204    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7prbx\" (UniqueName: \"kubernetes.io/projected/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-kube-api-access-7prbx\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.416814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.416783    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.418694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.418675    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.436317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.436284    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7prbx\" (UniqueName: \"kubernetes.io/projected/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-kube-api-access-7prbx\") pod \"maas-api-revoke-sub-67fdb213-2k68g\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.539032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.538995    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:01.854878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:01.854842    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"]
Oct 30 13:30:01.856175 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:30:01.856147    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1a3415e_33ca_4d86_819c_ea4eb26eb3f8.slice/crio-fafcf00728974e929f7c0e797a60247427d56cc89ce8f3e65de15b2e4cfaee65 WatchSource:0}: Error finding container fafcf00728974e929f7c0e797a60247427d56cc89ce8f3e65de15b2e4cfaee65: Status 404 returned error can't find the container with id fafcf00728974e929f7c0e797a60247427d56cc89ce8f3e65de15b2e4cfaee65
Oct 30 13:30:02.238619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.238585    4291 generic.go:356] "Generic (PLEG): container finished" podID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerID="8a71354dc951fb8d11dbb595cc836b73a203535aac55b070028472b1fd6068ba" exitCode=60
Oct 30 13:30:02.239088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.238686    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" event={"ID":"9f98cd42-d637-4f52-8e4c-54447615de8d","Type":"ContainerDied","Data":"8a71354dc951fb8d11dbb595cc836b73a203535aac55b070028472b1fd6068ba"}
Oct 30 13:30:02.239088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.239025    4291 scope.go:122] "RemoveContainer" containerID="8a71354dc951fb8d11dbb595cc836b73a203535aac55b070028472b1fd6068ba"
Oct 30 13:30:02.240665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.240515    4291 generic.go:356] "Generic (PLEG): container finished" podID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerID="006a4b5db64cb921321e054ac472d235876385116843e37ac2047a69cacaab94" exitCode=60
Oct 30 13:30:02.240665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.240584    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" event={"ID":"c4a553d3-8e02-4ab2-b0f6-044873b091b7","Type":"ContainerDied","Data":"006a4b5db64cb921321e054ac472d235876385116843e37ac2047a69cacaab94"}
Oct 30 13:30:02.240890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.240844    4291 scope.go:122] "RemoveContainer" containerID="006a4b5db64cb921321e054ac472d235876385116843e37ac2047a69cacaab94"
Oct 30 13:30:02.242359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.242332    4291 generic.go:356] "Generic (PLEG): container finished" podID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerID="c8160ab8cf619ec0816ec141f7ea9f549e55edf52776ae27cb63038070edbc24" exitCode=60
Oct 30 13:30:02.242440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.242419    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerDied","Data":"c8160ab8cf619ec0816ec141f7ea9f549e55edf52776ae27cb63038070edbc24"}
Oct 30 13:30:02.242668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.242629    4291 scope.go:122] "RemoveContainer" containerID="c8160ab8cf619ec0816ec141f7ea9f549e55edf52776ae27cb63038070edbc24"
Oct 30 13:30:02.244092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.244070    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g" event={"ID":"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8","Type":"ContainerStarted","Data":"2e6f178be130ce87068a0216d4aa0cb862c88162940ea28e66ca19becd1a1410"}
Oct 30 13:30:02.244168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.244122    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g" event={"ID":"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8","Type":"ContainerStarted","Data":"fafcf00728974e929f7c0e797a60247427d56cc89ce8f3e65de15b2e4cfaee65"}
Oct 30 13:30:02.246236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.246215    4291 generic.go:356] "Generic (PLEG): container finished" podID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerID="d625bce7de5df28db51de8b9c93bec48b1314b68730779a2fdebfe84290581e8" exitCode=60
Oct 30 13:30:02.246353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.246295    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" event={"ID":"18e5dcb1-b9b2-4ed3-8115-96945314789e","Type":"ContainerDied","Data":"d625bce7de5df28db51de8b9c93bec48b1314b68730779a2fdebfe84290581e8"}
Oct 30 13:30:02.246527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.246511    4291 scope.go:122] "RemoveContainer" containerID="d625bce7de5df28db51de8b9c93bec48b1314b68730779a2fdebfe84290581e8"
Oct 30 13:30:02.248060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.248036    4291 generic.go:356] "Generic (PLEG): container finished" podID="533b7741-937a-461f-8030-04da2f8af69c" containerID="fc7fcb7dac35f3258bcae0549851b691d5e8afb150d36a187d67fb6cb4068fae" exitCode=60
Oct 30 13:30:02.248161 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.248097    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" event={"ID":"533b7741-937a-461f-8030-04da2f8af69c","Type":"ContainerDied","Data":"fc7fcb7dac35f3258bcae0549851b691d5e8afb150d36a187d67fb6cb4068fae"}
Oct 30 13:30:02.248387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.248369    4291 scope.go:122] "RemoveContainer" containerID="fc7fcb7dac35f3258bcae0549851b691d5e8afb150d36a187d67fb6cb4068fae"
Oct 30 13:30:02.360358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:02.360302    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g" podStartSLOduration=1.36028588 podStartE2EDuration="1.36028588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:02.354547432 +0000 UTC m=+1612.614785667" watchObservedRunningTime="2026-10-30 13:30:02.36028588 +0000 UTC m=+1612.620524114"
Oct 30 13:30:03.260395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.260362    4291 generic.go:356] "Generic (PLEG): container finished" podID="c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" containerID="2e6f178be130ce87068a0216d4aa0cb862c88162940ea28e66ca19becd1a1410" exitCode=0
Oct 30 13:30:03.260745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.260439    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g" event={"ID":"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8","Type":"ContainerDied","Data":"2e6f178be130ce87068a0216d4aa0cb862c88162940ea28e66ca19becd1a1410"}
Oct 30 13:30:03.262378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.262353    4291 generic.go:356] "Generic (PLEG): container finished" podID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerID="4a33293a1623a6662f9dad8fcb193cda09c886398ea7e27862f660c384fa02ab" exitCode=60
Oct 30 13:30:03.262493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.262423    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" event={"ID":"18e5dcb1-b9b2-4ed3-8115-96945314789e","Type":"ContainerDied","Data":"4a33293a1623a6662f9dad8fcb193cda09c886398ea7e27862f660c384fa02ab"}
Oct 30 13:30:03.262493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.262475    4291 scope.go:122] "RemoveContainer" containerID="d625bce7de5df28db51de8b9c93bec48b1314b68730779a2fdebfe84290581e8"
Oct 30 13:30:03.262795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.262777    4291 scope.go:122] "RemoveContainer" containerID="4a33293a1623a6662f9dad8fcb193cda09c886398ea7e27862f660c384fa02ab"
Oct 30 13:30:03.263153 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:03.263127    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k_odh-ai-gateway-infra(18e5dcb1-b9b2-4ed3-8115-96945314789e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e"
Oct 30 13:30:03.264482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.264460    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" event={"ID":"533b7741-937a-461f-8030-04da2f8af69c","Type":"ContainerStarted","Data":"c7231df55f95f636cbfd699a897d6fad25bf6ff0c37de7ab6cd3c9be7f2d774f"}
Oct 30 13:30:03.266492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.266472    4291 generic.go:356] "Generic (PLEG): container finished" podID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerID="fd43dcfdb4bca19dc9efa527d6ce3c0d5e6f99b526accbf0612e56fc6c9829e3" exitCode=60
Oct 30 13:30:03.266591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.266562    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" event={"ID":"9f98cd42-d637-4f52-8e4c-54447615de8d","Type":"ContainerDied","Data":"fd43dcfdb4bca19dc9efa527d6ce3c0d5e6f99b526accbf0612e56fc6c9829e3"}
Oct 30 13:30:03.266793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.266771    4291 scope.go:122] "RemoveContainer" containerID="fd43dcfdb4bca19dc9efa527d6ce3c0d5e6f99b526accbf0612e56fc6c9829e3"
Oct 30 13:30:03.267091 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:03.267070    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb_odh-ai-gateway-infra(9f98cd42-d637-4f52-8e4c-54447615de8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d"
Oct 30 13:30:03.268656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.268621    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" event={"ID":"c4a553d3-8e02-4ab2-b0f6-044873b091b7","Type":"ContainerStarted","Data":"9d994021ea0b7484b1951ccc4d5b1e857030a3f7b8a009f4aa9ad19b97d8d675"}
Oct 30 13:30:03.270451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.270434    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerStarted","Data":"98025151b30ef6524c2014283bdd4b5cbd23e4fb54a21784ab0101361c93dc6a"}
Oct 30 13:30:03.303130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.303108    4291 scope.go:122] "RemoveContainer" containerID="8a71354dc951fb8d11dbb595cc836b73a203535aac55b070028472b1fd6068ba"
Oct 30 13:30:03.387617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.387560    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" podStartSLOduration=3.387547803 podStartE2EDuration="3.387547803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.385769772 +0000 UTC m=+1613.646008008" watchObservedRunningTime="2026-10-30 13:30:03.387547803 +0000 UTC m=+1613.647786036"
Oct 30 13:30:03.424408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:03.424351    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" podStartSLOduration=3.424337977 podStartE2EDuration="3.424337977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:03.421855891 +0000 UTC m=+1613.682094125" watchObservedRunningTime="2026-10-30 13:30:03.424337977 +0000 UTC m=+1613.684576210"
Oct 30 13:30:04.282425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.280965    4291 scope.go:122] "RemoveContainer" containerID="4a33293a1623a6662f9dad8fcb193cda09c886398ea7e27862f660c384fa02ab"
Oct 30 13:30:04.282425 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:04.281395    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k_odh-ai-gateway-infra(18e5dcb1-b9b2-4ed3-8115-96945314789e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e"
Oct 30 13:30:04.286185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.286157    4291 generic.go:356] "Generic (PLEG): container finished" podID="533b7741-937a-461f-8030-04da2f8af69c" containerID="c7231df55f95f636cbfd699a897d6fad25bf6ff0c37de7ab6cd3c9be7f2d774f" exitCode=60
Oct 30 13:30:04.286338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.286190    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" event={"ID":"533b7741-937a-461f-8030-04da2f8af69c","Type":"ContainerDied","Data":"c7231df55f95f636cbfd699a897d6fad25bf6ff0c37de7ab6cd3c9be7f2d774f"}
Oct 30 13:30:04.286338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.286256    4291 scope.go:122] "RemoveContainer" containerID="fc7fcb7dac35f3258bcae0549851b691d5e8afb150d36a187d67fb6cb4068fae"
Oct 30 13:30:04.286665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.286612    4291 scope.go:122] "RemoveContainer" containerID="c7231df55f95f636cbfd699a897d6fad25bf6ff0c37de7ab6cd3c9be7f2d774f"
Oct 30 13:30:04.287032 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:04.287012    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf_odh-ai-gateway-infra(533b7741-937a-461f-8030-04da2f8af69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" podUID="533b7741-937a-461f-8030-04da2f8af69c"
Oct 30 13:30:04.288481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.288464    4291 scope.go:122] "RemoveContainer" containerID="fd43dcfdb4bca19dc9efa527d6ce3c0d5e6f99b526accbf0612e56fc6c9829e3"
Oct 30 13:30:04.288862 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:04.288825    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb_odh-ai-gateway-infra(9f98cd42-d637-4f52-8e4c-54447615de8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d"
Oct 30 13:30:04.290245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.290224    4291 generic.go:356] "Generic (PLEG): container finished" podID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerID="9d994021ea0b7484b1951ccc4d5b1e857030a3f7b8a009f4aa9ad19b97d8d675" exitCode=60
Oct 30 13:30:04.290334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.290301    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" event={"ID":"c4a553d3-8e02-4ab2-b0f6-044873b091b7","Type":"ContainerDied","Data":"9d994021ea0b7484b1951ccc4d5b1e857030a3f7b8a009f4aa9ad19b97d8d675"}
Oct 30 13:30:04.290569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.290497    4291 scope.go:122] "RemoveContainer" containerID="9d994021ea0b7484b1951ccc4d5b1e857030a3f7b8a009f4aa9ad19b97d8d675"
Oct 30 13:30:04.290812 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:04.290790    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-wk4rz_odh-ai-gateway-infra(c4a553d3-8e02-4ab2-b0f6-044873b091b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7"
Oct 30 13:30:04.292304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.292281    4291 generic.go:356] "Generic (PLEG): container finished" podID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerID="98025151b30ef6524c2014283bdd4b5cbd23e4fb54a21784ab0101361c93dc6a" exitCode=60
Oct 30 13:30:04.292377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.292337    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerDied","Data":"98025151b30ef6524c2014283bdd4b5cbd23e4fb54a21784ab0101361c93dc6a"}
Oct 30 13:30:04.292586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.292572    4291 scope.go:122] "RemoveContainer" containerID="98025151b30ef6524c2014283bdd4b5cbd23e4fb54a21784ab0101361c93dc6a"
Oct 30 13:30:04.292886 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:04.292870    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9_odh-ai-gateway-infra(672131ae-497d-48e1-93f7-c1497cdd8f2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b"
Oct 30 13:30:04.300675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.300651    4291 scope.go:122] "RemoveContainer" containerID="006a4b5db64cb921321e054ac472d235876385116843e37ac2047a69cacaab94"
Oct 30 13:30:04.415721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.415695    4291 scope.go:122] "RemoveContainer" containerID="c8160ab8cf619ec0816ec141f7ea9f549e55edf52776ae27cb63038070edbc24"
Oct 30 13:30:04.548408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.548331    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:04.666422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.666383    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-openshift-ca\") pod \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") "
Oct 30 13:30:04.666622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.666488    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-kube-api-access-7prbx\" (UniqueName: \"kubernetes.io/projected/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-kube-api-access-7prbx\") pod \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") "
Oct 30 13:30:04.666622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.666565    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-xks-ca\") pod \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\" (UID: \"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8\") "
Oct 30 13:30:04.666849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.666819    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-openshift-ca" pod "c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" (UID: "c1a3415e-33ca-4d86-819c-ea4eb26eb3f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:04.667094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.667074    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:04.668859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.668824    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-kube-api-access-7prbx" pod "c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" (UID: "c1a3415e-33ca-4d86-819c-ea4eb26eb3f8"). InnerVolumeSpecName "kube-api-access-7prbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:04.668859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.668846    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-xks-ca" pod "c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" (UID: "c1a3415e-33ca-4d86-819c-ea4eb26eb3f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:30:04.768976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.768942    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7prbx\" (UniqueName: \"kubernetes.io/projected/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-kube-api-access-7prbx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:04.768976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:04.768972    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:05.301759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:05.301726    4291 scope.go:122] "RemoveContainer" containerID="c7231df55f95f636cbfd699a897d6fad25bf6ff0c37de7ab6cd3c9be7f2d774f"
Oct 30 13:30:05.302207 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:05.302058    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf_odh-ai-gateway-infra(533b7741-937a-461f-8030-04da2f8af69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" podUID="533b7741-937a-461f-8030-04da2f8af69c"
Oct 30 13:30:05.303443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:05.303422    4291 scope.go:122] "RemoveContainer" containerID="9d994021ea0b7484b1951ccc4d5b1e857030a3f7b8a009f4aa9ad19b97d8d675"
Oct 30 13:30:05.303714 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:05.303694    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-wk4rz_odh-ai-gateway-infra(c4a553d3-8e02-4ab2-b0f6-044873b091b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7"
Oct 30 13:30:05.304924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:05.304908    4291 scope.go:122] "RemoveContainer" containerID="98025151b30ef6524c2014283bdd4b5cbd23e4fb54a21784ab0101361c93dc6a"
Oct 30 13:30:05.305162 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:05.305143    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9_odh-ai-gateway-infra(672131ae-497d-48e1-93f7-c1497cdd8f2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b"
Oct 30 13:30:05.305945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:05.305928    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"
Oct 30 13:30:05.306026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:05.305927    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g" event={"ID":"c1a3415e-33ca-4d86-819c-ea4eb26eb3f8","Type":"ContainerDied","Data":"fafcf00728974e929f7c0e797a60247427d56cc89ce8f3e65de15b2e4cfaee65"}
Oct 30 13:30:05.306084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:05.306049    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fafcf00728974e929f7c0e797a60247427d56cc89ce8f3e65de15b2e4cfaee65"
Oct 30 13:30:08.211411 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:08.211371    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:30:11.161100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:11.161067    4291 scope.go:122] "RemoveContainer" containerID="0c5c39f4702f8649c16822165292aa05f2904f5f2fd16158dab72a10235b6c4b"
Oct 30 13:30:14.871258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:14.871220    4291 scope.go:122] "RemoveContainer" containerID="9d994021ea0b7484b1951ccc4d5b1e857030a3f7b8a009f4aa9ad19b97d8d675"
Oct 30 13:30:15.406127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:15.406091    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" event={"ID":"c4a553d3-8e02-4ab2-b0f6-044873b091b7","Type":"ContainerStarted","Data":"91e522d692c20b6da877930cccb8bcbf43df1aa56147c0859edc6bfde694702b"}
Oct 30 13:30:15.878695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:15.878661    4291 scope.go:122] "RemoveContainer" containerID="fd43dcfdb4bca19dc9efa527d6ce3c0d5e6f99b526accbf0612e56fc6c9829e3"
Oct 30 13:30:16.415868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.415827    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" event={"ID":"9f98cd42-d637-4f52-8e4c-54447615de8d","Type":"ContainerStarted","Data":"7f0f3309b730b587aca285b6bd8c0da756f7c15c2acb68838611116386535851"}
Oct 30 13:30:16.418070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.418041    4291 generic.go:356] "Generic (PLEG): container finished" podID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerID="91e522d692c20b6da877930cccb8bcbf43df1aa56147c0859edc6bfde694702b" exitCode=60
Oct 30 13:30:16.418070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.418071    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" event={"ID":"c4a553d3-8e02-4ab2-b0f6-044873b091b7","Type":"ContainerDied","Data":"91e522d692c20b6da877930cccb8bcbf43df1aa56147c0859edc6bfde694702b"}
Oct 30 13:30:16.418287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.418112    4291 scope.go:122] "RemoveContainer" containerID="9d994021ea0b7484b1951ccc4d5b1e857030a3f7b8a009f4aa9ad19b97d8d675"
Oct 30 13:30:16.418507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.418484    4291 scope.go:122] "RemoveContainer" containerID="91e522d692c20b6da877930cccb8bcbf43df1aa56147c0859edc6bfde694702b"
Oct 30 13:30:16.418947 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:16.418923    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889450-wk4rz_odh-ai-gateway-infra(c4a553d3-8e02-4ab2-b0f6-044873b091b7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7"
Oct 30 13:30:16.469932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.469897    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"]
Oct 30 13:30:16.471785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.471739    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" podStartSLOduration=16.471726867 podStartE2EDuration="16.471726867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:16.470825201 +0000 UTC m=+1626.731063435" watchObservedRunningTime="2026-10-30 13:30:16.471726867 +0000 UTC m=+1626.731965101"
Oct 30 13:30:16.871650 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:16.871615    4291 scope.go:122] "RemoveContainer" containerID="4a33293a1623a6662f9dad8fcb193cda09c886398ea7e27862f660c384fa02ab"
Oct 30 13:30:17.428710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.428665    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" event={"ID":"18e5dcb1-b9b2-4ed3-8115-96945314789e","Type":"ContainerStarted","Data":"9ecabf41fa051159121a0a5e0018001e8553607697e3cb13dc1f45a5ba7eaf79"}
Oct 30 13:30:17.430975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.430950    4291 generic.go:356] "Generic (PLEG): container finished" podID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerID="7f0f3309b730b587aca285b6bd8c0da756f7c15c2acb68838611116386535851" exitCode=60
Oct 30 13:30:17.431112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.431017    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" event={"ID":"9f98cd42-d637-4f52-8e4c-54447615de8d","Type":"ContainerDied","Data":"7f0f3309b730b587aca285b6bd8c0da756f7c15c2acb68838611116386535851"}
Oct 30 13:30:17.431112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.431073    4291 scope.go:122] "RemoveContainer" containerID="fd43dcfdb4bca19dc9efa527d6ce3c0d5e6f99b526accbf0612e56fc6c9829e3"
Oct 30 13:30:17.431412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.431395    4291 scope.go:122] "RemoveContainer" containerID="7f0f3309b730b587aca285b6bd8c0da756f7c15c2acb68838611116386535851"
Oct 30 13:30:17.431782 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:17.431758    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb_odh-ai-gateway-infra(9f98cd42-d637-4f52-8e4c-54447615de8d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d"
Oct 30 13:30:17.477411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.477361    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" podStartSLOduration=17.477345893 podStartE2EDuration="17.477345893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:17.477204985 +0000 UTC m=+1627.737443219" watchObservedRunningTime="2026-10-30 13:30:17.477345893 +0000 UTC m=+1627.737584126"
Oct 30 13:30:17.537411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.537366    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"]
Oct 30 13:30:17.604414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.604385    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:17.756616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.756537    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4a553d3-8e02-4ab2-b0f6-044873b091b7-kube-api-access-hnf8n\" (UniqueName: \"kubernetes.io/projected/c4a553d3-8e02-4ab2-b0f6-044873b091b7-kube-api-access-hnf8n\") pod \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") "
Oct 30 13:30:17.756821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.756625    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4a553d3-8e02-4ab2-b0f6-044873b091b7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a553d3-8e02-4ab2-b0f6-044873b091b7-maas-api-service-ca\") pod \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\" (UID: \"c4a553d3-8e02-4ab2-b0f6-044873b091b7\") "
Oct 30 13:30:17.757013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.756989    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4a553d3-8e02-4ab2-b0f6-044873b091b7-maas-api-service-ca" pod "c4a553d3-8e02-4ab2-b0f6-044873b091b7" (UID: "c4a553d3-8e02-4ab2-b0f6-044873b091b7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:17.757236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.757216    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4a553d3-8e02-4ab2-b0f6-044873b091b7-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.758770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.758742    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4a553d3-8e02-4ab2-b0f6-044873b091b7-kube-api-access-hnf8n" pod "c4a553d3-8e02-4ab2-b0f6-044873b091b7" (UID: "c4a553d3-8e02-4ab2-b0f6-044873b091b7"). InnerVolumeSpecName "kube-api-access-hnf8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:17.858125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.858091    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hnf8n\" (UniqueName: \"kubernetes.io/projected/c4a553d3-8e02-4ab2-b0f6-044873b091b7-kube-api-access-hnf8n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:17.872630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.872292    4291 scope.go:122] "RemoveContainer" containerID="c7231df55f95f636cbfd699a897d6fad25bf6ff0c37de7ab6cd3c9be7f2d774f"
Oct 30 13:30:17.873735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:17.873259    4291 scope.go:122] "RemoveContainer" containerID="98025151b30ef6524c2014283bdd4b5cbd23e4fb54a21784ab0101361c93dc6a"
Oct 30 13:30:18.442876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.442843    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"
Oct 30 13:30:18.443715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.443500    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz" event={"ID":"c4a553d3-8e02-4ab2-b0f6-044873b091b7","Type":"ContainerDied","Data":"d266531d2b0ce3dedbbf7a23e601753c4e4b3a2ac3a2161735d32a58cee39218"}
Oct 30 13:30:18.443715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.443569    4291 scope.go:122] "RemoveContainer" containerID="91e522d692c20b6da877930cccb8bcbf43df1aa56147c0859edc6bfde694702b"
Oct 30 13:30:18.447938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.447233    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerStarted","Data":"d5ddec1fdc14e6fff628176d9172df44d2eb30b5a4efc5667d309c2fb4aee34b"}
Oct 30 13:30:18.454648 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.454612    4291 generic.go:356] "Generic (PLEG): container finished" podID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerID="9ecabf41fa051159121a0a5e0018001e8553607697e3cb13dc1f45a5ba7eaf79" exitCode=60
Oct 30 13:30:18.454792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.454694    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" event={"ID":"18e5dcb1-b9b2-4ed3-8115-96945314789e","Type":"ContainerDied","Data":"9ecabf41fa051159121a0a5e0018001e8553607697e3cb13dc1f45a5ba7eaf79"}
Oct 30 13:30:18.455060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.455045    4291 scope.go:122] "RemoveContainer" containerID="9ecabf41fa051159121a0a5e0018001e8553607697e3cb13dc1f45a5ba7eaf79"
Oct 30 13:30:18.455341 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:18.455323    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k_odh-ai-gateway-infra(18e5dcb1-b9b2-4ed3-8115-96945314789e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e"
Oct 30 13:30:18.465838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.465810    4291 scope.go:122] "RemoveContainer" containerID="4a33293a1623a6662f9dad8fcb193cda09c886398ea7e27862f660c384fa02ab"
Oct 30 13:30:18.466053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.466022    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" event={"ID":"533b7741-937a-461f-8030-04da2f8af69c","Type":"ContainerStarted","Data":"3f5232e90925a0e4ebb5a8c99bcd18a762d155c2dc853a8d78fa3e63033db0de"}
Oct 30 13:30:18.491429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.491404    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"]
Oct 30 13:30:18.562291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.562254    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"]
Oct 30 13:30:18.571910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.571879    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889450-wk4rz"]
Oct 30 13:30:18.635316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.635290    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:18.681166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681131    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"]
Oct 30 13:30:18.681708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681690    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.681708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681710    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681732    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681740    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681752    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681760    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681782    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681788    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681801    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681808    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681818    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681826    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681858    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" containerName="revoke-keys"
Oct 30 13:30:18.681882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.681865    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" containerName="revoke-keys"
Oct 30 13:30:18.682458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.682031    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" containerName="revoke-keys"
Oct 30 13:30:18.682458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.682046    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.682458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.682057    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:18.682458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.682068    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.682458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.682077    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.682458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.682086    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" containerName="cleanup"
Oct 30 13:30:18.685583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.685559    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:18.763752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.758593    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-f45f04-dockercfg-26pn6\""
Oct 30 13:30:18.765105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.765035    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"]
Oct 30 13:30:18.829223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.827928    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-f45f04\""
Oct 30 13:30:18.829223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.828670    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f98cd42-d637-4f52-8e4c-54447615de8d-kube-api-access-dh5h7\" (UniqueName: \"kubernetes.io/projected/9f98cd42-d637-4f52-8e4c-54447615de8d-kube-api-access-dh5h7\") pod \"9f98cd42-d637-4f52-8e4c-54447615de8d\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") "
Oct 30 13:30:18.829223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.828717    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f98cd42-d637-4f52-8e4c-54447615de8d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f98cd42-d637-4f52-8e4c-54447615de8d-maas-api-service-ca\") pod \"9f98cd42-d637-4f52-8e4c-54447615de8d\" (UID: \"9f98cd42-d637-4f52-8e4c-54447615de8d\") "
Oct 30 13:30:18.829223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.828979    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da0687-9a51-4371-9287-d56f620ac584-plugins-config-volume\") pod \"payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") " pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:18.829223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.829091    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wtgl\" (UniqueName: \"kubernetes.io/projected/c0da0687-9a51-4371-9287-d56f620ac584-kube-api-access-7wtgl\") pod \"payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") " pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:18.829781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.829731    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f98cd42-d637-4f52-8e4c-54447615de8d-maas-api-service-ca" pod "9f98cd42-d637-4f52-8e4c-54447615de8d" (UID: "9f98cd42-d637-4f52-8e4c-54447615de8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:18.841112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.841038    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f98cd42-d637-4f52-8e4c-54447615de8d-kube-api-access-dh5h7" pod "9f98cd42-d637-4f52-8e4c-54447615de8d" (UID: "9f98cd42-d637-4f52-8e4c-54447615de8d"). InnerVolumeSpecName "kube-api-access-dh5h7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:18.866952 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:18.866922    4291 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672131ae_497d_48e1_93f7_c1497cdd8f2b.slice/crio-d5ddec1fdc14e6fff628176d9172df44d2eb30b5a4efc5667d309c2fb4aee34b.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 13:30:18.930769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.930729    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da0687-9a51-4371-9287-d56f620ac584-plugins-config-volume\") pod \"payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") " pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:18.930976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.930843    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wtgl\" (UniqueName: \"kubernetes.io/projected/c0da0687-9a51-4371-9287-d56f620ac584-kube-api-access-7wtgl\") pod \"payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") " pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:18.930976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.930934    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh5h7\" (UniqueName: \"kubernetes.io/projected/9f98cd42-d637-4f52-8e4c-54447615de8d-kube-api-access-dh5h7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:18.930976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.930950    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9f98cd42-d637-4f52-8e4c-54447615de8d-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:18.931493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.931472    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da0687-9a51-4371-9287-d56f620ac584-plugins-config-volume\") pod \"payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") " pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:18.964419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:18.964393    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wtgl\" (UniqueName: \"kubernetes.io/projected/c0da0687-9a51-4371-9287-d56f620ac584-kube-api-access-7wtgl\") pod \"payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") " pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:19.111537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.111501    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:30:19.134121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.133902    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:19.483184 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.483039    4291 generic.go:356] "Generic (PLEG): container finished" podID="533b7741-937a-461f-8030-04da2f8af69c" containerID="3f5232e90925a0e4ebb5a8c99bcd18a762d155c2dc853a8d78fa3e63033db0de" exitCode=60
Oct 30 13:30:19.483184 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.483135    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" event={"ID":"533b7741-937a-461f-8030-04da2f8af69c","Type":"ContainerDied","Data":"3f5232e90925a0e4ebb5a8c99bcd18a762d155c2dc853a8d78fa3e63033db0de"}
Oct 30 13:30:19.483602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.483222    4291 scope.go:122] "RemoveContainer" containerID="c7231df55f95f636cbfd699a897d6fad25bf6ff0c37de7ab6cd3c9be7f2d774f"
Oct 30 13:30:19.483749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.483663    4291 scope.go:122] "RemoveContainer" containerID="3f5232e90925a0e4ebb5a8c99bcd18a762d155c2dc853a8d78fa3e63033db0de"
Oct 30 13:30:19.484714 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:19.484047    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf_odh-ai-gateway-infra(533b7741-937a-461f-8030-04da2f8af69c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" podUID="533b7741-937a-461f-8030-04da2f8af69c"
Oct 30 13:30:19.485172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.485155    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"
Oct 30 13:30:19.485605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.485179    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb" event={"ID":"9f98cd42-d637-4f52-8e4c-54447615de8d","Type":"ContainerDied","Data":"781d9363996566dc3220c1ed534e24d77582a24cb309fd54953c8aba64efc756"}
Oct 30 13:30:19.486298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.486279    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"]
Oct 30 13:30:19.488055 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:30:19.488031    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0da0687_9a51_4371_9287_d56f620ac584.slice/crio-99ff6a76a91b38914b79f3c3b9dc84dad8a78271b9c217687b5714cc4208e8e0 WatchSource:0}: Error finding container 99ff6a76a91b38914b79f3c3b9dc84dad8a78271b9c217687b5714cc4208e8e0: Status 404 returned error can't find the container with id 99ff6a76a91b38914b79f3c3b9dc84dad8a78271b9c217687b5714cc4208e8e0
Oct 30 13:30:19.493155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.493127    4291 generic.go:356] "Generic (PLEG): container finished" podID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerID="d5ddec1fdc14e6fff628176d9172df44d2eb30b5a4efc5667d309c2fb4aee34b" exitCode=60
Oct 30 13:30:19.493267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.493204    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerDied","Data":"d5ddec1fdc14e6fff628176d9172df44d2eb30b5a4efc5667d309c2fb4aee34b"}
Oct 30 13:30:19.493646 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.493596    4291 scope.go:122] "RemoveContainer" containerID="d5ddec1fdc14e6fff628176d9172df44d2eb30b5a4efc5667d309c2fb4aee34b"
Oct 30 13:30:19.493992 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:19.493937    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9_odh-ai-gateway-infra(672131ae-497d-48e1-93f7-c1497cdd8f2b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b"
Oct 30 13:30:19.506835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.506807    4291 scope.go:122] "RemoveContainer" containerID="7f0f3309b730b587aca285b6bd8c0da756f7c15c2acb68838611116386535851"
Oct 30 13:30:19.517927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.517739    4291 scope.go:122] "RemoveContainer" containerID="98025151b30ef6524c2014283bdd4b5cbd23e4fb54a21784ab0101361c93dc6a"
Oct 30 13:30:19.540968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.540946    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"]
Oct 30 13:30:19.601715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.601683    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"]
Oct 30 13:30:19.605797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.605768    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5852ff-29889450-hv2fb"]
Oct 30 13:30:19.650746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.650718    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:19.675600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.675567    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"]
Oct 30 13:30:19.865656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.854263    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e5dcb1-b9b2-4ed3-8115-96945314789e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e5dcb1-b9b2-4ed3-8115-96945314789e-maas-api-service-ca\") pod \"18e5dcb1-b9b2-4ed3-8115-96945314789e\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") "
Oct 30 13:30:19.865656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.854374    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e5dcb1-b9b2-4ed3-8115-96945314789e-kube-api-access-9wfwx\" (UniqueName: \"kubernetes.io/projected/18e5dcb1-b9b2-4ed3-8115-96945314789e-kube-api-access-9wfwx\") pod \"18e5dcb1-b9b2-4ed3-8115-96945314789e\" (UID: \"18e5dcb1-b9b2-4ed3-8115-96945314789e\") "
Oct 30 13:30:19.869506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.869466    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e5dcb1-b9b2-4ed3-8115-96945314789e-maas-api-service-ca" pod "18e5dcb1-b9b2-4ed3-8115-96945314789e" (UID: "18e5dcb1-b9b2-4ed3-8115-96945314789e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:19.963192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.962668    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18e5dcb1-b9b2-4ed3-8115-96945314789e-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:19.970132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.969809    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" path="/var/lib/kubelet/pods/9f98cd42-d637-4f52-8e4c-54447615de8d/volumes"
Oct 30 13:30:19.970515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.970428    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4a553d3-8e02-4ab2-b0f6-044873b091b7" path="/var/lib/kubelet/pods/c4a553d3-8e02-4ab2-b0f6-044873b091b7/volumes"
Oct 30 13:30:19.970515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:19.970464    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e5dcb1-b9b2-4ed3-8115-96945314789e-kube-api-access-9wfwx" pod "18e5dcb1-b9b2-4ed3-8115-96945314789e" (UID: "18e5dcb1-b9b2-4ed3-8115-96945314789e"). InnerVolumeSpecName "kube-api-access-9wfwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:20.064252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.064203    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wfwx\" (UniqueName: \"kubernetes.io/projected/18e5dcb1-b9b2-4ed3-8115-96945314789e-kube-api-access-9wfwx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:20.508584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.508473    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"
Oct 30 13:30:20.509711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.508422    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k" event={"ID":"18e5dcb1-b9b2-4ed3-8115-96945314789e","Type":"ContainerDied","Data":"7e23f637494672fea552d785b7247ffef9221d91bd9be54cb0cc2d3aeff7c422"}
Oct 30 13:30:20.509856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.509750    4291 scope.go:122] "RemoveContainer" containerID="9ecabf41fa051159121a0a5e0018001e8553607697e3cb13dc1f45a5ba7eaf79"
Oct 30 13:30:20.516555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.516523    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq" event={"ID":"c0da0687-9a51-4371-9287-d56f620ac584","Type":"ContainerStarted","Data":"0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916"}
Oct 30 13:30:20.516762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.516742    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq" event={"ID":"c0da0687-9a51-4371-9287-d56f620ac584","Type":"ContainerStarted","Data":"99ff6a76a91b38914b79f3c3b9dc84dad8a78271b9c217687b5714cc4208e8e0"}
Oct 30 13:30:20.517738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.517717    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:20.596832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.593476    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq" podStartSLOduration=2.593453315 podStartE2EDuration="2.593453315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:20.580152413 +0000 UTC m=+1630.840390650" watchObservedRunningTime="2026-10-30 13:30:20.593453315 +0000 UTC m=+1630.853691551"
Oct 30 13:30:20.735087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.735060    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"]
Oct 30 13:30:20.736342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:20.736306    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-f9ebb5-29889450-9r76k"]
Oct 30 13:30:21.171861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.171322    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:21.174009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.173917    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:21.327878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.317871    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672131ae-497d-48e1-93f7-c1497cdd8f2b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672131ae-497d-48e1-93f7-c1497cdd8f2b-maas-api-service-ca\") pod \"672131ae-497d-48e1-93f7-c1497cdd8f2b\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") "
Oct 30 13:30:21.327878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.326172    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/533b7741-937a-461f-8030-04da2f8af69c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b7741-937a-461f-8030-04da2f8af69c-maas-api-service-ca\") pod \"533b7741-937a-461f-8030-04da2f8af69c\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") "
Oct 30 13:30:21.327878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.326298    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/533b7741-937a-461f-8030-04da2f8af69c-kube-api-access-bd9nn\" (UniqueName: \"kubernetes.io/projected/533b7741-937a-461f-8030-04da2f8af69c-kube-api-access-bd9nn\") pod \"533b7741-937a-461f-8030-04da2f8af69c\" (UID: \"533b7741-937a-461f-8030-04da2f8af69c\") "
Oct 30 13:30:21.327878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.326325    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672131ae-497d-48e1-93f7-c1497cdd8f2b-kube-api-access-7q5n8\" (UniqueName: \"kubernetes.io/projected/672131ae-497d-48e1-93f7-c1497cdd8f2b-kube-api-access-7q5n8\") pod \"672131ae-497d-48e1-93f7-c1497cdd8f2b\" (UID: \"672131ae-497d-48e1-93f7-c1497cdd8f2b\") "
Oct 30 13:30:21.327878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.318544    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672131ae-497d-48e1-93f7-c1497cdd8f2b-maas-api-service-ca" pod "672131ae-497d-48e1-93f7-c1497cdd8f2b" (UID: "672131ae-497d-48e1-93f7-c1497cdd8f2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.327878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.327444    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/533b7741-937a-461f-8030-04da2f8af69c-maas-api-service-ca" pod "533b7741-937a-461f-8030-04da2f8af69c" (UID: "533b7741-937a-461f-8030-04da2f8af69c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:30:21.428208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.428170    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/533b7741-937a-461f-8030-04da2f8af69c-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.428433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.428415    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672131ae-497d-48e1-93f7-c1497cdd8f2b-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.456029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.455826    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/533b7741-937a-461f-8030-04da2f8af69c-kube-api-access-bd9nn" pod "533b7741-937a-461f-8030-04da2f8af69c" (UID: "533b7741-937a-461f-8030-04da2f8af69c"). InnerVolumeSpecName "kube-api-access-bd9nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.505666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.494592    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672131ae-497d-48e1-93f7-c1497cdd8f2b-kube-api-access-7q5n8" pod "672131ae-497d-48e1-93f7-c1497cdd8f2b" (UID: "672131ae-497d-48e1-93f7-c1497cdd8f2b"). InnerVolumeSpecName "kube-api-access-7q5n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:30:21.564660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.541661    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd9nn\" (UniqueName: \"kubernetes.io/projected/533b7741-937a-461f-8030-04da2f8af69c-kube-api-access-bd9nn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.565261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.565232    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q5n8\" (UniqueName: \"kubernetes.io/projected/672131ae-497d-48e1-93f7-c1497cdd8f2b-kube-api-access-7q5n8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:30:21.882267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.882237    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"
Oct 30 13:30:21.889876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.889843    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" path="/var/lib/kubelet/pods/18e5dcb1-b9b2-4ed3-8115-96945314789e/volumes"
Oct 30 13:30:21.896204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.896178    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf" event={"ID":"533b7741-937a-461f-8030-04da2f8af69c","Type":"ContainerDied","Data":"dfc0417b732b1971fff5123bd795804439f0b3eadb8587144370f3877af5c158"}
Oct 30 13:30:21.896374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.896360    4291 scope.go:122] "RemoveContainer" containerID="3f5232e90925a0e4ebb5a8c99bcd18a762d155c2dc853a8d78fa3e63033db0de"
Oct 30 13:30:21.913568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.913529    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"
Oct 30 13:30:21.919860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.919819    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9" event={"ID":"672131ae-497d-48e1-93f7-c1497cdd8f2b","Type":"ContainerDied","Data":"28823a1e594dbfc6439f0dc888635329d0f944188f45eb5a5f6b596a29bbbbc8"}
Oct 30 13:30:21.934440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.934318    4291 scope.go:122] "RemoveContainer" containerID="d5ddec1fdc14e6fff628176d9172df44d2eb30b5a4efc5667d309c2fb4aee34b"
Oct 30 13:30:21.991763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.991689    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"]
Oct 30 13:30:21.998088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:21.998050    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889450-7q5wf"]
Oct 30 13:30:22.034404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:22.034369    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"]
Oct 30 13:30:22.042730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:22.042693    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889450-qkzw9"]
Oct 30 13:30:23.879947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:23.879914    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="533b7741-937a-461f-8030-04da2f8af69c" path="/var/lib/kubelet/pods/533b7741-937a-461f-8030-04da2f8af69c/volumes"
Oct 30 13:30:23.880450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:23.880373    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" path="/var/lib/kubelet/pods/672131ae-497d-48e1-93f7-c1497cdd8f2b/volumes"
Oct 30 13:30:28.038842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:28.038804    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"]
Oct 30 13:30:28.050912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:28.050877    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b651407-w2n5g"]
Oct 30 13:30:29.879309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:29.879269    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd7a72e-185d-4d33-a64c-d40b2edf97e3" path="/var/lib/kubelet/pods/abd7a72e-185d-4d33-a64c-d40b2edf97e3/volumes"
Oct 30 13:30:32.924390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:32.924359    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:30:38.212289 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:30:38.212242    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:30:40.945750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.945717    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"]
Oct 30 13:30:40.946197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946183    4291 cpu_manager.go:414] "containerMap: removing container" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946198    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946210    4291 cpu_manager.go:414] "containerMap: removing container" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946215    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946224    4291 cpu_manager.go:414] "containerMap: removing container" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946229    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946236    4291 cpu_manager.go:414] "containerMap: removing container" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946241    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946248    4291 cpu_manager.go:414] "containerMap: removing container" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946252    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946266    4291 cpu_manager.go:414] "containerMap: removing container" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946270    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946286    4291 cpu_manager.go:414] "containerMap: removing container" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946290    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946368    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946375    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946381    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f98cd42-d637-4f52-8e4c-54447615de8d" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946387    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946393    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946401    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:40.946494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.946409    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:40.950078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.950057    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:40.988897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:40.988864    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"]
Oct 30 13:30:41.111302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111267    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111367    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111427    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111461    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111611    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111672    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgbld\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-kube-api-access-rgbld\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111815    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111843    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.111924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.111907    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.213492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.213402    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.213492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.213445    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.213492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.213484    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.213816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.213509    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.213816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.213556    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.213816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.213580    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgbld\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-kube-api-access-rgbld\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.213978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.213922    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.214041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.214002    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.214107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.214089    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.214160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.214132    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.214215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.214159    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.214279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.214215    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.214502 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.214478    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.214584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.214504    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.216022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.215998    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.216144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.216127    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.237833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.237799    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgbld\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-kube-api-access-rgbld\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.238012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.237993    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.268128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.268093    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:41.606439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.606414    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"]
Oct 30 13:30:41.612415 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:30:41.609347    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdff5ab_53ed_4c01_9858_d9f97611dbbf.slice/crio-0e956b59190de15b355075bef3b358c7ad2e3502da24adc499cd4ed860b86240 WatchSource:0}: Error finding container 0e956b59190de15b355075bef3b358c7ad2e3502da24adc499cd4ed860b86240: Status 404 returned error can't find the container with id 0e956b59190de15b355075bef3b358c7ad2e3502da24adc499cd4ed860b86240
Oct 30 13:30:41.616515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.616469    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:41.616625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.616567    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:41.616625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:41.616618    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:42.078085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:42.078004    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" event={"ID":"0cdff5ab-53ed-4c01-9858-d9f97611dbbf","Type":"ContainerStarted","Data":"a20baaf687e71cf58e1e1710972e314913f7d7e0b79704215983041dfe20063c"}
Oct 30 13:30:42.078085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:42.078062    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" event={"ID":"0cdff5ab-53ed-4c01-9858-d9f97611dbbf","Type":"ContainerStarted","Data":"0e956b59190de15b355075bef3b358c7ad2e3502da24adc499cd4ed860b86240"}
Oct 30 13:30:42.149828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:42.149771    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podStartSLOduration=2.14975736 podStartE2EDuration="2.14975736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:42.137542875 +0000 UTC m=+1652.397781110" watchObservedRunningTime="2026-10-30 13:30:42.14975736 +0000 UTC m=+1652.409995593"
Oct 30 13:30:42.268709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:42.268665    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:42.270308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:42.270286    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:30:42.270380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:42.270348    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 13:30:43.271452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:43.271416    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:30:43.272127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:43.272096    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 13:30:44.275091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:44.275048    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:30:44.275532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:44.275122    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 13:30:44.697591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:44.697550    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:30:45.272659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.271052    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:30:45.272659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.271120    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 13:30:45.818909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.818861    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"]
Oct 30 13:30:45.819432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.819377    4291 cpu_manager.go:414] "containerMap: removing container" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:45.819432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.819389    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:45.819432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.819403    4291 cpu_manager.go:414] "containerMap: removing container" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:45.819432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.819408    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:45.819709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.819515    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="533b7741-937a-461f-8030-04da2f8af69c" containerName="cleanup"
Oct 30 13:30:45.819709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.819534    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e5dcb1-b9b2-4ed3-8115-96945314789e" containerName="cleanup"
Oct 30 13:30:45.819709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.819545    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="672131ae-497d-48e1-93f7-c1497cdd8f2b" containerName="cleanup"
Oct 30 13:30:45.825857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.825828    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:45.856926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:45.850839    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"]
Oct 30 13:30:46.022735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.022699    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.023037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.023017    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.023167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.023149    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.023310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.023295    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.023425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.023409    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.023606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.023588    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h98fn\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-kube-api-access-h98fn\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.023761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.023744    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.023912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.023896    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.024058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.024041    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125022    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125072    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125133    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125213    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125241    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125433    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-credential-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125492    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-data\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125538    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-socket\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125543    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125587    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125706    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h98fn\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-kube-api-access-h98fn\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125757    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.125965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125810    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-certs\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.126022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.125993    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istiod-ca-cert\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.128153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.128115    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-envoy\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.128375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.128353    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-podinfo\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.146481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.146452    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-token\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.146666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.146621    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h98fn\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-kube-api-access-h98fn\") pod \"e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") " pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.281661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.281123    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:46.290198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.288763    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:30:46.290198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.288824    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 13:30:46.808727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.808614    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"]
Oct 30 13:30:46.821344 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:30:46.821304    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20686636_cecc_4cba_80f7_0d1d4bd8e1b7.slice/crio-7569f094f30dfaf31fc9ad6d877f19b3afabf36d5de2aa79d83110a72d38f22e WatchSource:0}: Error finding container 7569f094f30dfaf31fc9ad6d877f19b3afabf36d5de2aa79d83110a72d38f22e: Status 404 returned error can't find the container with id 7569f094f30dfaf31fc9ad6d877f19b3afabf36d5de2aa79d83110a72d38f22e
Oct 30 13:30:46.828814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.825683    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:46.828814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.825777    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:46.828814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:46.825820    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:47.271092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:47.270527    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:30:47.271092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:47.270606    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 13:30:47.277657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:47.274076    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" event={"ID":"20686636-cecc-4cba-80f7-0d1d4bd8e1b7","Type":"ContainerStarted","Data":"0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8"}
Oct 30 13:30:47.277991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:47.277964    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" event={"ID":"20686636-cecc-4cba-80f7-0d1d4bd8e1b7","Type":"ContainerStarted","Data":"7569f094f30dfaf31fc9ad6d877f19b3afabf36d5de2aa79d83110a72d38f22e"}
Oct 30 13:30:47.294900 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:47.294871    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:47.309579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:47.308917    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:47.309579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:47.308994    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:48.269416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:48.269375    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused" start-of-body=
Oct 30 13:30:48.270009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:48.269920    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.168:15021/healthz/ready\": dial tcp 10.133.0.168:15021: connect: connection refused"
Oct 30 13:30:48.282710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:48.282673    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:48.282912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:48.282740    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:49.282798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:49.282758    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:49.283414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:49.283381    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:49.847440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:49.847405    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:49.847966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:49.847934    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:49.848885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:49.848867    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:30:49.933248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:49.933189    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podStartSLOduration=4.933168784 podStartE2EDuration="4.933168784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:47.402181076 +0000 UTC m=+1657.662419312" watchObservedRunningTime="2026-10-30 13:30:49.933168784 +0000 UTC m=+1660.193407019"
Oct 30 13:30:50.282314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:50.282224    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:50.282314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:50.282285    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:51.281793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:51.281758    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:51.282223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:51.281827    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:52.282213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:52.282176    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:52.282792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:52.282241    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:53.282149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:53.282053    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:53.282149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:53.282115    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:54.282227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:54.282184    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body=
Oct 30 13:30:54.282416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:54.282255    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused"
Oct 30 13:30:55.287507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:55.287475    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:55.287991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:55.287932    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:55.288665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:55.288619    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:30:55.943024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:55.942984    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"]
Oct 30 13:30:55.948357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:55.948332    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:55.968390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:55.968355    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"]
Oct 30 13:30:56.040899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.040860    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.040926    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.041002    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.041073    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.041175    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.041240    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.041276    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.041305    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.041450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.041431    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhfg4\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-kube-api-access-zhfg4\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143087    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhfg4\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-kube-api-access-zhfg4\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143140    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143174    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143202    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143278    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143364    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143449    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143496    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143538    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143709    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143764    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-data\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143808    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-certs\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.143981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.143885    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-credential-socket\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.144287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.144264    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istiod-ca-cert\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.145503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.145485    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-envoy\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.145913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.145893    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-podinfo\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.163669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.163620    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-token\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.163820 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.163701    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhfg4\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-kube-api-access-zhfg4\") pod \"e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") " pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.265014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.264932    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:56.599219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.599188    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"]
Oct 30 13:30:56.599754 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:30:56.599732    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9196df4b_d6bc_4ed9_8a50_dc8f828d48b3.slice/crio-61a9a02f06f0f1b0fa01afc7dba3cb433d515635d2773378102f1c7b27b1341e WatchSource:0}: Error finding container 61a9a02f06f0f1b0fa01afc7dba3cb433d515635d2773378102f1c7b27b1341e: Status 404 returned error can't find the container with id 61a9a02f06f0f1b0fa01afc7dba3cb433d515635d2773378102f1c7b27b1341e
Oct 30 13:30:56.602334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.602301    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:56.602408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.602373    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:56.602408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:56.602401    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:30:57.384578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:57.384543    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" event={"ID":"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3","Type":"ContainerStarted","Data":"458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15"}
Oct 30 13:30:57.384808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:57.384599    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" event={"ID":"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3","Type":"ContainerStarted","Data":"61a9a02f06f0f1b0fa01afc7dba3cb433d515635d2773378102f1c7b27b1341e"}
Oct 30 13:30:57.440302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:57.440234    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podStartSLOduration=2.440213683 podStartE2EDuration="2.440213683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:30:57.43705046 +0000 UTC m=+1667.697288695" watchObservedRunningTime="2026-10-30 13:30:57.440213683 +0000 UTC m=+1667.700451918"
Oct 30 13:30:58.274309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:58.274272    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:30:58.276546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:58.276511    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:30:58.276696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:58.276573    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:30:59.273658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:59.273131    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:30:59.273658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:30:59.273196    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:31:00.267180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:00.267134    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:31:00.267882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:00.267838    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:31:01.033973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:01.033937    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"]
Oct 30 13:31:01.045448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:01.045416    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa8dd3b-dmqzw"]
Oct 30 13:31:01.266886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:01.266844    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:31:01.267198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:01.267161    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:31:01.886245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:01.886197    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29c90f38-8cb2-4158-9535-e276c3b8703b" path="/var/lib/kubelet/pods/29c90f38-8cb2-4158-9535-e276c3b8703b/volumes"
Oct 30 13:31:02.280716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:02.274059    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:31:02.280716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:02.274149    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:31:03.266421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:03.266329    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused" start-of-body=
Oct 30 13:31:03.266601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:03.266418    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.170:15021/healthz/ready\": dial tcp 10.133.0.170:15021: connect: connection refused"
Oct 30 13:31:04.270261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:04.270231    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:31:04.270747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:04.270732    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:31:04.271486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:04.271461    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:31:08.212738 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:31:08.212693    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:31:09.986409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:09.986372    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:31:11.207056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:11.207016    4291 scope.go:122] "RemoveContainer" containerID="a402e5c6a07e026dcc7c9b02153e032b817f05534dbb53a66b9f62ccc2383a4a"
Oct 30 13:31:11.221474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:11.221445    4291 scope.go:122] "RemoveContainer" containerID="7263aeb703d72cce83bc693af58ec21000d7349519361eaa4df8d7f7272d5a92"
Oct 30 13:31:11.234549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:11.234519    4291 scope.go:122] "RemoveContainer" containerID="312859af663593eb18c07dad75769ef787c77169f0d4522477204d4798b2326b"
Oct 30 13:31:11.245571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:11.245543    4291 scope.go:122] "RemoveContainer" containerID="06f20999915b2456cedc76ad1ac9c511d6822fcf2f49e5902a175dc4d812a81a"
Oct 30 13:31:11.257858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:11.257839    4291 scope.go:122] "RemoveContainer" containerID="9fb4cdd60f8e1458573e23385332de051b5a0ab7f00845f77e1f4950def9f28b"
Oct 30 13:31:13.222506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:13.222474    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"]
Oct 30 13:31:13.222943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:13.222668    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy" containerID="cri-o://1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb" gracePeriod=30
Oct 30 13:31:18.033696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.033658    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"]
Oct 30 13:31:18.046032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.045998    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f8ff967-pz78l"]
Oct 30 13:31:18.509728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.509704    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:31:18.559083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.559051    4291 generic.go:356] "Generic (PLEG): container finished" podID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerID="1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb" exitCode=0
Oct 30 13:31:18.559270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.559106    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"
Oct 30 13:31:18.559270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.559141    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" event={"ID":"1611a92a-f72d-4bae-b03a-fa1b14777d11","Type":"ContainerDied","Data":"1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb"}
Oct 30 13:31:18.559270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.559198    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n" event={"ID":"1611a92a-f72d-4bae-b03a-fa1b14777d11","Type":"ContainerDied","Data":"f892c7212f2b5aa09ea3e13319f9f88fbddd434ab9a4ae9adfb3532d34043484"}
Oct 30 13:31:18.559270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.559220    4291 scope.go:122] "RemoveContainer" containerID="1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb"
Oct 30 13:31:18.570473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.570435    4291 scope.go:122] "RemoveContainer" containerID="1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb"
Oct 30 13:31:18.570867 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:31:18.570841    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb\": container with ID starting with 1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb not found: ID does not exist" containerID="1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb"
Oct 30 13:31:18.570952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.570880    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb"} err="failed to get container status \"1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb\": rpc error: code = NotFound desc = could not find container \"1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb\": container with ID starting with 1a4f4cd4db48d367c2d2988b806d1a14f784c0c6ed2226ad20dfee809a42b9bb not found: ID does not exist"
Oct 30 13:31:18.636952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.636912    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-credential-socket\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637014    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-kube-api-access-qtfp7\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-kube-api-access-qtfp7\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637049    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-socket\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637129    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-podinfo\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637171    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-token\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-token\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637208    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-certs\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637216    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-credential-socket" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:18.637320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637279    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1611a92a-f72d-4bae-b03a-fa1b14777d11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611a92a-f72d-4bae-b03a-fa1b14777d11-istiod-ca-cert\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637366    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-data\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637383    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-socket" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:18.637606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637465    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-envoy\") pod \"1611a92a-f72d-4bae-b03a-fa1b14777d11\" (UID: \"1611a92a-f72d-4bae-b03a-fa1b14777d11\") "
Oct 30 13:31:18.637606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637537    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-certs" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:18.637934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637846    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-data" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:18.637934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.637878    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1611a92a-f72d-4bae-b03a-fa1b14777d11-istiod-ca-cert" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:31:18.638050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.638016    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.638050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.638032    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1611a92a-f72d-4bae-b03a-fa1b14777d11-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.638050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.638044    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.638280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.638056    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.638280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.638071    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.639754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.639724    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-token" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:18.639899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.639820    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-podinfo" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:31:18.639899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.639852    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-kube-api-access-qtfp7" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "kube-api-access-qtfp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:31:18.640029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.640008    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-envoy" pod "1611a92a-f72d-4bae-b03a-fa1b14777d11" (UID: "1611a92a-f72d-4bae-b03a-fa1b14777d11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:31:18.742367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.739518    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.742367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.739560    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfp7\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-kube-api-access-qtfp7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.742367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.739575    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.742367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.739589    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1611a92a-f72d-4bae-b03a-fa1b14777d11-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:31:18.912768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.912669    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"]
Oct 30 13:31:18.922586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:18.922552    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-svl2n"]
Oct 30 13:31:19.501769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.501721    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"]
Oct 30 13:31:19.502724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.502700    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy"
Oct 30 13:31:19.502724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.502723    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy"
Oct 30 13:31:19.502913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.502833    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" containerName="istio-proxy"
Oct 30 13:31:19.508090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.508060    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.528399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.528361    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"]
Oct 30 13:31:19.535270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.535241    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-29a3f3-openshift-default-dockercfg-2sfrp\""
Oct 30 13:31:19.650660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.650591    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-data\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.650884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.650670    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.650884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.650724    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-istio-token\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.650884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.650813    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3397ccd9-392f-4245-a839-48fd4da8d393-istio-podinfo\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.651078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.650943    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkdms\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-kube-api-access-hkdms\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.651078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.651028    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3397ccd9-392f-4245-a839-48fd4da8d393-istiod-ca-cert\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.669718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.651069    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-certs\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.669718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.651113    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-envoy\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.669718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.651196    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-credential-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.752750    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3397ccd9-392f-4245-a839-48fd4da8d393-istio-podinfo\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.752879    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkdms\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-kube-api-access-hkdms\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.752964    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3397ccd9-392f-4245-a839-48fd4da8d393-istiod-ca-cert\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.753007    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-certs\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.753041    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-envoy\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.753106    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-credential-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.753189    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-data\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.753227    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.759704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.753265    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-istio-token\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.779658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.779550    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3397ccd9-392f-4245-a839-48fd4da8d393-istiod-ca-cert\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.780397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.780068    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-credential-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.780397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.780187    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-socket\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.780397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.780361    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-certs\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.780699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.780675    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-data\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.807176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.807112    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-envoy\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.811011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.807783    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-istio-token\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.811011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.810471    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkdms\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-kube-api-access-hkdms\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.812506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.812477    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3397ccd9-392f-4245-a839-48fd4da8d393-istio-podinfo\") pod \"e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") " pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.827935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.827895    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:19.888720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.886387    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1611a92a-f72d-4bae-b03a-fa1b14777d11" path="/var/lib/kubelet/pods/1611a92a-f72d-4bae-b03a-fa1b14777d11/volumes"
Oct 30 13:31:19.888720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:19.886970    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="957f86af-96e7-4bc6-b534-e1e166bf8392" path="/var/lib/kubelet/pods/957f86af-96e7-4bc6-b534-e1e166bf8392/volumes"
Oct 30 13:31:20.175491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.175450    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"]
Oct 30 13:31:20.179828 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:31:20.179786    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3397ccd9_392f_4245_a839_48fd4da8d393.slice/crio-4ba8388ca43f065537145ccb908070fa48b57ed70a068cd4f5d20ea479bb6d5c WatchSource:0}: Error finding container 4ba8388ca43f065537145ccb908070fa48b57ed70a068cd4f5d20ea479bb6d5c: Status 404 returned error can't find the container with id 4ba8388ca43f065537145ccb908070fa48b57ed70a068cd4f5d20ea479bb6d5c
Oct 30 13:31:20.183007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.182972    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:20.183120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.183063    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:20.183120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.183109    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:20.584783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.584673    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" event={"ID":"3397ccd9-392f-4245-a839-48fd4da8d393","Type":"ContainerStarted","Data":"33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1"}
Oct 30 13:31:20.584783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.584741    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" event={"ID":"3397ccd9-392f-4245-a839-48fd4da8d393","Type":"ContainerStarted","Data":"4ba8388ca43f065537145ccb908070fa48b57ed70a068cd4f5d20ea479bb6d5c"}
Oct 30 13:31:20.628895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.628843    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" podStartSLOduration=1.628823801 podStartE2EDuration="1.628823801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:20.625255715 +0000 UTC m=+1690.885493962" watchObservedRunningTime="2026-10-30 13:31:20.628823801 +0000 UTC m=+1690.889062034"
Oct 30 13:31:20.828112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:20.828077    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:21.835895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:21.835861    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:22.599857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:22.599826    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:22.600964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:22.600942    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:31:25.185047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.185007    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"]
Oct 30 13:31:25.189122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.189092    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.206826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.206794    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"]
Oct 30 13:31:25.214144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.214108    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-29a3f3\""
Oct 30 13:31:25.214348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.214204    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-29a3f3-dockercfg-xsv5w\""
Oct 30 13:31:25.338555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.338507    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e231fd7-3743-4341-83ba-61583a6461fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.338800 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.338575    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-269ms\" (UniqueName: \"kubernetes.io/projected/6e231fd7-3743-4341-83ba-61583a6461fb-kube-api-access-269ms\") pod \"payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.342993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.342954    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"]
Oct 30 13:31:25.348852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.348818    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.366375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.366339    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"]
Oct 30 13:31:25.447723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.442447    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e231fd7-3743-4341-83ba-61583a6461fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.447723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.442507    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-269ms\" (UniqueName: \"kubernetes.io/projected/6e231fd7-3743-4341-83ba-61583a6461fb-kube-api-access-269ms\") pod \"payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.447723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.442590    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6xxh\" (UniqueName: \"kubernetes.io/projected/cd9eb694-130e-4dcd-a814-9487970b78d2-kube-api-access-s6xxh\") pod \"payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.447723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.442717    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb694-130e-4dcd-a814-9487970b78d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.472660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.450057    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e231fd7-3743-4341-83ba-61583a6461fb-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.501735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.501269    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-269ms\" (UniqueName: \"kubernetes.io/projected/6e231fd7-3743-4341-83ba-61583a6461fb-kube-api-access-269ms\") pod \"payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.505042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.504998    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:25.548027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.546437    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6xxh\" (UniqueName: \"kubernetes.io/projected/cd9eb694-130e-4dcd-a814-9487970b78d2-kube-api-access-s6xxh\") pod \"payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.548027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.546555    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb694-130e-4dcd-a814-9487970b78d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.548027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.547553    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb694-130e-4dcd-a814-9487970b78d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.573655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.573013    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6xxh\" (UniqueName: \"kubernetes.io/projected/cd9eb694-130e-4dcd-a814-9487970b78d2-kube-api-access-s6xxh\") pod \"payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.684756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.684713    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:25.846142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.846111    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"]
Oct 30 13:31:25.846621 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:31:25.846599    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e231fd7_3743_4341_83ba_61583a6461fb.slice/crio-488cac855ed29603d85ad3e6d16380ba90ea64a2de4ed14df052cb367416eeb1 WatchSource:0}: Error finding container 488cac855ed29603d85ad3e6d16380ba90ea64a2de4ed14df052cb367416eeb1: Status 404 returned error can't find the container with id 488cac855ed29603d85ad3e6d16380ba90ea64a2de4ed14df052cb367416eeb1
Oct 30 13:31:25.993845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:25.993814    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"]
Oct 30 13:31:25.994376 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:31:25.994352    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd9eb694_130e_4dcd_a814_9487970b78d2.slice/crio-dad1ea4c19a34b970225f8f02706dc58dbb437969776a716deb0fbcb38f301c0 WatchSource:0}: Error finding container dad1ea4c19a34b970225f8f02706dc58dbb437969776a716deb0fbcb38f301c0: Status 404 returned error can't find the container with id dad1ea4c19a34b970225f8f02706dc58dbb437969776a716deb0fbcb38f301c0
Oct 30 13:31:26.639765 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.639723    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" event={"ID":"6e231fd7-3743-4341-83ba-61583a6461fb","Type":"ContainerStarted","Data":"61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8"}
Oct 30 13:31:26.640286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.639831    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" event={"ID":"6e231fd7-3743-4341-83ba-61583a6461fb","Type":"ContainerStarted","Data":"488cac855ed29603d85ad3e6d16380ba90ea64a2de4ed14df052cb367416eeb1"}
Oct 30 13:31:26.640286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.640039    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:26.641708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.641681    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" event={"ID":"cd9eb694-130e-4dcd-a814-9487970b78d2","Type":"ContainerStarted","Data":"3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe"}
Oct 30 13:31:26.641836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.641727    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" event={"ID":"cd9eb694-130e-4dcd-a814-9487970b78d2","Type":"ContainerStarted","Data":"dad1ea4c19a34b970225f8f02706dc58dbb437969776a716deb0fbcb38f301c0"}
Oct 30 13:31:26.641836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.641755    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:26.712728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.712624    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" podStartSLOduration=1.71260429 podStartE2EDuration="1.71260429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:26.711059341 +0000 UTC m=+1696.971297576" watchObservedRunningTime="2026-10-30 13:31:26.71260429 +0000 UTC m=+1696.972842526"
Oct 30 13:31:26.772062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:26.771996    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" podStartSLOduration=1.771972954 podStartE2EDuration="1.771972954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:26.770590729 +0000 UTC m=+1697.030828967" watchObservedRunningTime="2026-10-30 13:31:26.771972954 +0000 UTC m=+1697.032211187"
Oct 30 13:31:27.611682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.611644    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"]
Oct 30 13:31:27.616128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.616102    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.637502 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.637467    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"]
Oct 30 13:31:27.644163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.644128    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-c684d2-openshift-default-dockercfg-f677f\""
Oct 30 13:31:27.775989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.775950    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.775989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.775990    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.776260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.776115    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.776260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.776163    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.776260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.776218    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gx77m\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-kube-api-access-gx77m\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.776260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.776241    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.776468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.776367    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcce731-6465-4cf0-9b99-7e3748a6818a-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.776468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.776440    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.776716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.776614    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878211    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878340    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878377    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878404    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878488    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878529    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878566    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gx77m\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-kube-api-access-gx77m\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878595    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878697    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcce731-6465-4cf0-9b99-7e3748a6818a-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.878961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.878865    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-data\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.879437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.879413    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcce731-6465-4cf0-9b99-7e3748a6818a-istiod-ca-cert\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.879518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.879421    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-certs\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.881400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.881367    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-envoy\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.881666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.881610    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.881759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.881743    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-podinfo\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.881879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.881860    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-credential-socket\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.897895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.897850    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-token\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.898039 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.897861    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gx77m\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-kube-api-access-gx77m\") pod \"e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") " pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:27.931787 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:27.931755    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:28.258925 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:31:28.258896    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfcce731_6465_4cf0_9b99_7e3748a6818a.slice/crio-0d12a906bdadd431681791011706cbe82142888206363603248ba138965903e8 WatchSource:0}: Error finding container 0d12a906bdadd431681791011706cbe82142888206363603248ba138965903e8: Status 404 returned error can't find the container with id 0d12a906bdadd431681791011706cbe82142888206363603248ba138965903e8
Oct 30 13:31:28.259050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.258987    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"]
Oct 30 13:31:28.262135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.262100    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:28.262229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.262186    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:28.262269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.262228    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:28.663956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.663914    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" event={"ID":"cfcce731-6465-4cf0-9b99-7e3748a6818a","Type":"ContainerStarted","Data":"a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38"}
Oct 30 13:31:28.664369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.663991    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" event={"ID":"cfcce731-6465-4cf0-9b99-7e3748a6818a","Type":"ContainerStarted","Data":"0d12a906bdadd431681791011706cbe82142888206363603248ba138965903e8"}
Oct 30 13:31:28.706999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.706940    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" podStartSLOduration=1.706924965 podStartE2EDuration="1.706924965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:28.703838231 +0000 UTC m=+1698.964076466" watchObservedRunningTime="2026-10-30 13:31:28.706924965 +0000 UTC m=+1698.967163200"
Oct 30 13:31:28.936715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:28.936222    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:29.934476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:29.934441    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:31:29.935005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:29.934494    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:31:30.739371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.739340    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"]
Oct 30 13:31:30.744017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.743989    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.796240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.796214    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"]
Oct 30 13:31:30.803383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.803348    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-167965\"/\"openshift-service-ca.crt\""
Oct 30 13:31:30.803505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.803462    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-167965\"/\"kube-root-ca.crt\""
Oct 30 13:31:30.808918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.803586    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-167965\"/\"auth-test-model-167965-kserve-self-signed-certs\""
Oct 30 13:31:30.808918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.803766    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-167965\"/\"default-dockercfg-sn7mf\""
Oct 30 13:31:30.824348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.824315    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-model-cache\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.824500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.824380    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-dshm\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.824573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.824490    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-home\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.824699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.824678    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tmp-dir\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.824759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.824712    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tls-certs\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.824798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.824787    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k972k\" (UniqueName: \"kubernetes.io/projected/85b0d393-40b4-45d4-bb8f-2b13d049c78a-kube-api-access-k972k\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.925734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.925693    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k972k\" (UniqueName: \"kubernetes.io/projected/85b0d393-40b4-45d4-bb8f-2b13d049c78a-kube-api-access-k972k\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.925913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.925771    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-model-cache\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.925913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.925837    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-dshm\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.925913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.925901    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-home\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.926171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.926064    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tmp-dir\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.926171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.926105    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tls-certs\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.926302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.926234    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-model-cache\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.926527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.926504    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tmp-dir\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.926626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.926550    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-home\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.928461 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.928430    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-dshm\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.928602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.928583    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tls-certs\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:30.932794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.932764    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:31:30.932927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.932817    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:31:30.947204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:30.947174    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k972k\" (UniqueName: \"kubernetes.io/projected/85b0d393-40b4-45d4-bb8f-2b13d049c78a-kube-api-access-k972k\") pod \"auth-test-model-167965-kserve-55c895896f-bh67r\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") " pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:31.104657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:31.102831    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:31.526616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:31.526576    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"]
Oct 30 13:31:31.529694 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:31:31.529664    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod85b0d393_40b4_45d4_bb8f_2b13d049c78a.slice/crio-1ccfdabe4971d8dd64712e2d872574473658f2b636f8889f55e0f65dcc952733 WatchSource:0}: Error finding container 1ccfdabe4971d8dd64712e2d872574473658f2b636f8889f55e0f65dcc952733: Status 404 returned error can't find the container with id 1ccfdabe4971d8dd64712e2d872574473658f2b636f8889f55e0f65dcc952733
Oct 30 13:31:31.699066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:31.699029    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r" event={"ID":"85b0d393-40b4-45d4-bb8f-2b13d049c78a","Type":"ContainerStarted","Data":"137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01"}
Oct 30 13:31:31.699223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:31.699085    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r" event={"ID":"85b0d393-40b4-45d4-bb8f-2b13d049c78a","Type":"ContainerStarted","Data":"1ccfdabe4971d8dd64712e2d872574473658f2b636f8889f55e0f65dcc952733"}
Oct 30 13:31:31.742295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:31.742238    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r" podStartSLOduration=1.742221198 podStartE2EDuration="1.742221198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:31.736073619 +0000 UTC m=+1701.996311854" watchObservedRunningTime="2026-10-30 13:31:31.742221198 +0000 UTC m=+1702.002459487"
Oct 30 13:31:31.932440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:31.932398    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:31:31.932649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:31.932496    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:31:32.104116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:32.104077    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:32.932895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:32.932867    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:31:32.933062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:32.932916    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:31:33.035107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:33.035072    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"]
Oct 30 13:31:33.046309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:33.046279    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d379a7e7-zwcqw"]
Oct 30 13:31:33.885497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:33.885449    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ab3645-5504-445f-a078-0a070972b4f2" path="/var/lib/kubelet/pods/74ab3645-5504-445f-a078-0a070972b4f2/volumes"
Oct 30 13:31:33.932851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:33.932811    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body=
Oct 30 13:31:33.933024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:33.932871    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused"
Oct 30 13:31:34.993506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:34.993443    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:34.994188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:34.994163    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:34.995399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:34.995373    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 13:31:37.656048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:37.656020    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 13:31:37.656582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:37.656085    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 13:31:38.210881 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:31:38.210848    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:31:40.146806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:40.146770    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:31:41.103940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:41.103899    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:41.117587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:41.117556    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:41.791173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:41.791145    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:31:48.048659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:48.048605    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"]
Oct 30 13:31:48.057721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:48.057690    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce0a3b3a-27wtw"]
Oct 30 13:31:49.878879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:49.878843    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="790da168-9e8a-4750-bd87-46dccb75570f" path="/var/lib/kubelet/pods/790da168-9e8a-4750-bd87-46dccb75570f/volumes"
Oct 30 13:31:51.131157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.131123    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"]
Oct 30 13:31:51.136946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.136925    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.159156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.159131    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"]
Oct 30 13:31:51.167362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.167334    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-cab6c5-openshift-default-dockercfg-dsbh2\""
Oct 30 13:31:51.289306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289272    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-token\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289310    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289328    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-data\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289438    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-credential-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289473    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-certs\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289495    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467869-5fad-4417-9f4f-d09c3c496e0d-istiod-ca-cert\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289610    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-envoy\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289674    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-podinfo\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.289765 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.289726    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtmpk\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-kube-api-access-rtmpk\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391110    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-envoy\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391154    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-podinfo\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391186    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtmpk\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-kube-api-access-rtmpk\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391302    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-token\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391325    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391349    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-data\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391403    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-credential-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391442    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-certs\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391727    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391761    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467869-5fad-4417-9f4f-d09c3c496e0d-istiod-ca-cert\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391795    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-data\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391845    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-credential-socket\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.391918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.391871    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-certs\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.392376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.392352    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467869-5fad-4417-9f4f-d09c3c496e0d-istiod-ca-cert\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.393716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.393686    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-envoy\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.393818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.393801    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-podinfo\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.409872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.409850    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-token\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.409962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.409918    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtmpk\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-kube-api-access-rtmpk\") pod \"e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") " pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.452147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.452127    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:51.776684 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.776650    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"]
Oct 30 13:31:51.776835 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:31:51.776802    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5467869_5fad_4417_9f4f_d09c3c496e0d.slice/crio-1d7c61907a4ab08c86dce6ae659cfb3b3d7d744acb6706221889c9150eb08d33 WatchSource:0}: Error finding container 1d7c61907a4ab08c86dce6ae659cfb3b3d7d744acb6706221889c9150eb08d33: Status 404 returned error can't find the container with id 1d7c61907a4ab08c86dce6ae659cfb3b3d7d744acb6706221889c9150eb08d33
Oct 30 13:31:51.780119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.780080    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:51.780213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.780168    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:51.780255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.780212    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:31:51.867238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.867200    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" event={"ID":"b5467869-5fad-4417-9f4f-d09c3c496e0d","Type":"ContainerStarted","Data":"a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5"}
Oct 30 13:31:51.867380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.867266    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" event={"ID":"b5467869-5fad-4417-9f4f-d09c3c496e0d","Type":"ContainerStarted","Data":"1d7c61907a4ab08c86dce6ae659cfb3b3d7d744acb6706221889c9150eb08d33"}
Oct 30 13:31:51.913964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:51.913800    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podStartSLOduration=0.913781064 podStartE2EDuration="913.781064ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:31:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:31:51.911325672 +0000 UTC m=+1722.171563907" watchObservedRunningTime="2026-10-30 13:31:51.913781064 +0000 UTC m=+1722.174019298"
Oct 30 13:31:52.461660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:52.456959    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:52.473661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:52.467762    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:31:52.473661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:52.467881    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:31:53.467660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:53.454006    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:31:53.467660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:53.454080    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:31:54.458664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:54.454002    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:31:54.458664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:54.454066    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:31:55.459661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:55.459081    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:31:55.459661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:55.459148    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:31:56.469661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:56.463071    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:31:56.469661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:56.463139    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:31:57.467497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:57.467052    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body=
Oct 30 13:31:57.467497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:57.467118    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused"
Oct 30 13:31:58.032311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:58.032270    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"]
Oct 30 13:31:58.043958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:58.043927    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85e3a53a-578lc"]
Oct 30 13:31:58.458543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:58.458503    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:58.458934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:58.458915    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:58.459730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:58.459709    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:31:59.884430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:31:59.884393    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="908e07ce-0c83-4d02-8a5a-f10fe785ac5e" path="/var/lib/kubelet/pods/908e07ce-0c83-4d02-8a5a-f10fe785ac5e/volumes"
Oct 30 13:32:08.211663 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:32:08.211605    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:32:11.424704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:11.424670    4291 scope.go:122] "RemoveContainer" containerID="36bac5979a1ffe7e4e0ced81102ec4ae5e8bd558b293796c4cebde54a285bb9d"
Oct 30 13:32:11.449181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:11.449154    4291 scope.go:122] "RemoveContainer" containerID="ff55d5d706ca10ad1702bbac1e790f6f0a31fa3fdd8a1d23aa906383be21197e"
Oct 30 13:32:11.491628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:11.491601    4291 scope.go:122] "RemoveContainer" containerID="b621c1cbfaec9117542e0a450d95934bef94a8494aa644b92cc78cc41e791f12"
Oct 30 13:32:12.036288 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:12.036254    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"]
Oct 30 13:32:12.053685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:12.049871    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-388faeb7-xl5tz"]
Oct 30 13:32:13.885939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:13.885904    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffd7e93c-2d21-4666-ac42-ad67543de96c" path="/var/lib/kubelet/pods/ffd7e93c-2d21-4666-ac42-ad67543de96c/volumes"
Oct 30 13:32:15.384922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.384879    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"]
Oct 30 13:32:15.391916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.391890    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.405462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.405428    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"]
Oct 30 13:32:15.411809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.411774    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:32:15.411978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.411860    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:32:15.411978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.411958    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:32:15.419178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.419149    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:32:15.456195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.456147    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.456752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.456729    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.457081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.457044    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqs6\" (UniqueName: \"kubernetes.io/projected/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-kube-api-access-5zqs6\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.558253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.558218    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.558451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.558284    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqs6\" (UniqueName: \"kubernetes.io/projected/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-kube-api-access-5zqs6\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.558451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.558354    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.559107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.559081    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.560921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.560886    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.577869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.577835    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqs6\" (UniqueName: \"kubernetes.io/projected/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-kube-api-access-5zqs6\") pod \"maas-api-revoke-keys-e2e-autores-f45f04-s5kv2\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:15.738731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:15.738618    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:16.051162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:16.051130    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"]
Oct 30 13:32:16.054603 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:32:16.054566    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc637ab26_9d39_4ab8_b1a6_32e0280e5eb4.slice/crio-04115be95cb999a9a05cfc910f8d8f8de80d062bd02bc9098903ff01a0960ada WatchSource:0}: Error finding container 04115be95cb999a9a05cfc910f8d8f8de80d062bd02bc9098903ff01a0960ada: Status 404 returned error can't find the container with id 04115be95cb999a9a05cfc910f8d8f8de80d062bd02bc9098903ff01a0960ada
Oct 30 13:32:16.164682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:16.164629    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2" event={"ID":"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4","Type":"ContainerStarted","Data":"04115be95cb999a9a05cfc910f8d8f8de80d062bd02bc9098903ff01a0960ada"}
Oct 30 13:32:17.175721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:17.175687    4291 generic.go:356] "Generic (PLEG): container finished" podID="c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" containerID="76b976b67194b88329cf0859f206217b08ea78e42521ab75272c474d8f1638c4" exitCode=0
Oct 30 13:32:17.176135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:17.175781    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2" event={"ID":"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4","Type":"ContainerDied","Data":"76b976b67194b88329cf0859f206217b08ea78e42521ab75272c474d8f1638c4"}
Oct 30 13:32:18.186562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.186529    4291 generic.go:356] "Generic (PLEG): container finished" podID="3397ccd9-392f-4245-a839-48fd4da8d393" containerID="33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1" exitCode=0
Oct 30 13:32:18.187046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.186603    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" event={"ID":"3397ccd9-392f-4245-a839-48fd4da8d393","Type":"ContainerDied","Data":"33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1"}
Oct 30 13:32:18.187095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.187072    4291 scope.go:122] "RemoveContainer" containerID="33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1"
Oct 30 13:32:18.189182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.189145    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:32:18.189291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.189250    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:32:18.189331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.189304    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:32:18.439178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.437554    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:18.505166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.505124    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-xks-ca\") pod \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") "
Oct 30 13:32:18.505426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.505406    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-kube-api-access-5zqs6\" (UniqueName: \"kubernetes.io/projected/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-kube-api-access-5zqs6\") pod \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") "
Oct 30 13:32:18.505515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.505474    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-openshift-ca\") pod \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\" (UID: \"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4\") "
Oct 30 13:32:18.505912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.505879    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-openshift-ca" pod "c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" (UID: "c637ab26-9d39-4ab8-b1a6-32e0280e5eb4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:18.506218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.506191    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:32:18.508601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.508568    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-kube-api-access-5zqs6" pod "c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" (UID: "c637ab26-9d39-4ab8-b1a6-32e0280e5eb4"). InnerVolumeSpecName "kube-api-access-5zqs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:18.511746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.511716    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-xks-ca" pod "c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" (UID: "c637ab26-9d39-4ab8-b1a6-32e0280e5eb4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:32:18.607384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.607351    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:32:18.607384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:18.607381    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zqs6\" (UniqueName: \"kubernetes.io/projected/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4-kube-api-access-5zqs6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:32:19.033090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.033054    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"]
Oct 30 13:32:19.050020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.049987    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20ced213-x7mrt"]
Oct 30 13:32:19.213106 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.213070    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" event={"ID":"3397ccd9-392f-4245-a839-48fd4da8d393","Type":"ContainerStarted","Data":"7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69"}
Oct 30 13:32:19.215219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.214879    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"
Oct 30 13:32:19.215219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.214901    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2" event={"ID":"c637ab26-9d39-4ab8-b1a6-32e0280e5eb4","Type":"ContainerDied","Data":"04115be95cb999a9a05cfc910f8d8f8de80d062bd02bc9098903ff01a0960ada"}
Oct 30 13:32:19.215219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.214953    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04115be95cb999a9a05cfc910f8d8f8de80d062bd02bc9098903ff01a0960ada"
Oct 30 13:32:19.828840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.828804    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:32:19.830325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.830291    4291 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:32:19.830464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.830361    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:32:19.938211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:19.937877    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae2751f1-0212-4ff2-a299-51f84a9013b5" path="/var/lib/kubelet/pods/ae2751f1-0212-4ff2-a299-51f84a9013b5/volumes"
Oct 30 13:32:20.828858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:20.828824    4291 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:32:20.829265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:20.828881    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:32:21.829193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:21.829152    4291 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:32:21.829713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:21.829216    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:32:22.600694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:22.600659    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:32:22.829059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:22.828978    4291 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:32:22.829059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:22.829036    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:32:23.037196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:23.037100    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"]
Oct 30 13:32:23.048675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:23.048628    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6fc2635-rsl7l"]
Oct 30 13:32:23.828458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:23.828424    4291 patch_prober.go:29] interesting pod/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused" start-of-body=
Oct 30 13:32:23.828670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:23.828479    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.171:15021/healthz/ready\": dial tcp 10.133.0.171:15021: connect: connection refused"
Oct 30 13:32:23.884045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:23.884012    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7" path="/var/lib/kubelet/pods/dc1cce30-bbe1-4c9d-aae5-fd717fcd7ca7/volumes"
Oct 30 13:32:24.833540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:24.833508    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:32:24.834603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:24.834579    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 13:32:25.764765 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:25.764722    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"]
Oct 30 13:32:26.318760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:26.318722    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:32:30.765873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:30.765829    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq" podUID="c0da0687-9a51-4371-9287-d56f620ac584" containerName="payload-processing" containerID="cri-o://0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916" gracePeriod=25
Oct 30 13:32:31.062592 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.062562    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:32:31.071751 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.071715    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0da0687-9a51-4371-9287-d56f620ac584-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da0687-9a51-4371-9287-d56f620ac584-plugins-config-volume\") pod \"c0da0687-9a51-4371-9287-d56f620ac584\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") "
Oct 30 13:32:31.071944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.071858    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0da0687-9a51-4371-9287-d56f620ac584-kube-api-access-7wtgl\" (UniqueName: \"kubernetes.io/projected/c0da0687-9a51-4371-9287-d56f620ac584-kube-api-access-7wtgl\") pod \"c0da0687-9a51-4371-9287-d56f620ac584\" (UID: \"c0da0687-9a51-4371-9287-d56f620ac584\") "
Oct 30 13:32:31.072223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.072192    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0da0687-9a51-4371-9287-d56f620ac584-plugins-config-volume" pod "c0da0687-9a51-4371-9287-d56f620ac584" (UID: "c0da0687-9a51-4371-9287-d56f620ac584"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:32:31.072413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.072385    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0da0687-9a51-4371-9287-d56f620ac584-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:32:31.074442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.074404    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0da0687-9a51-4371-9287-d56f620ac584-kube-api-access-7wtgl" pod "c0da0687-9a51-4371-9287-d56f620ac584" (UID: "c0da0687-9a51-4371-9287-d56f620ac584"). InnerVolumeSpecName "kube-api-access-7wtgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:32:31.173843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.173808    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wtgl\" (UniqueName: \"kubernetes.io/projected/c0da0687-9a51-4371-9287-d56f620ac584-kube-api-access-7wtgl\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:32:31.358677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.358615    4291 generic.go:356] "Generic (PLEG): container finished" podID="c0da0687-9a51-4371-9287-d56f620ac584" containerID="0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916" exitCode=0
Oct 30 13:32:31.358876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.358669    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq" event={"ID":"c0da0687-9a51-4371-9287-d56f620ac584","Type":"ContainerDied","Data":"0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916"}
Oct 30 13:32:31.358876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.358733    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq" event={"ID":"c0da0687-9a51-4371-9287-d56f620ac584","Type":"ContainerDied","Data":"99ff6a76a91b38914b79f3c3b9dc84dad8a78271b9c217687b5714cc4208e8e0"}
Oct 30 13:32:31.358876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.358756    4291 scope.go:122] "RemoveContainer" containerID="0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916"
Oct 30 13:32:31.358876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.358685    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"
Oct 30 13:32:31.370844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.370821    4291 scope.go:122] "RemoveContainer" containerID="0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916"
Oct 30 13:32:31.371221 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:32:31.371178    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916\": container with ID starting with 0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916 not found: ID does not exist" containerID="0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916"
Oct 30 13:32:31.371221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.371212    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916"} err="failed to get container status \"0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916\": rpc error: code = NotFound desc = could not find container \"0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916\": container with ID starting with 0e64f52fa83b88a72325738ae7f5e43ab05ff9dc7ab1650f90cd6ecb3541b916 not found: ID does not exist"
Oct 30 13:32:31.410049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.410010    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"]
Oct 30 13:32:31.422578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.422541    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-f45f04-748d55b9db-zhmtq"]
Oct 30 13:32:31.879584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:31.879549    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0da0687-9a51-4371-9287-d56f620ac584" path="/var/lib/kubelet/pods/c0da0687-9a51-4371-9287-d56f620ac584/volumes"
Oct 30 13:32:36.035896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:36.035857    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"]
Oct 30 13:32:36.047608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:36.047573    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d4b54114-xrvlg"]
Oct 30 13:32:37.038400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:37.038367    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"]
Oct 30 13:32:37.041948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:37.041920    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d487447-wgc4r"]
Oct 30 13:32:37.881912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:37.881873    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c36da58-217f-4527-ae20-86b987f254c6" path="/var/lib/kubelet/pods/3c36da58-217f-4527-ae20-86b987f254c6/volumes"
Oct 30 13:32:37.882616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:37.882582    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07c9487-95d2-4da5-b8f2-a3f0e46f33f1" path="/var/lib/kubelet/pods/d07c9487-95d2-4da5-b8f2-a3f0e46f33f1/volumes"
Oct 30 13:32:38.212268 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:32:38.212177    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:32:51.299595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:51.299558    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"]
Oct 30 13:32:51.311215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:51.311183    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-f45f04-s5kv2"]
Oct 30 13:32:51.879459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:32:51.879422    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" path="/var/lib/kubelet/pods/c637ab26-9d39-4ab8-b1a6-32e0280e5eb4/volumes"
Oct 30 13:33:01.038665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:01.038613    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"]
Oct 30 13:33:01.048124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:01.048093    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f9b336c-hd8wl"]
Oct 30 13:33:01.906509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:01.906469    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f50c5c52-6fb8-4c5c-81b5-4242707a1da8" path="/var/lib/kubelet/pods/f50c5c52-6fb8-4c5c-81b5-4242707a1da8/volumes"
Oct 30 13:33:05.857795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.857765    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"]
Oct 30 13:33:05.858357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.858343    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c0da0687-9a51-4371-9287-d56f620ac584" containerName="payload-processing"
Oct 30 13:33:05.858401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.858358    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0da0687-9a51-4371-9287-d56f620ac584" containerName="payload-processing"
Oct 30 13:33:05.858401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.858392    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" containerName="revoke-keys"
Oct 30 13:33:05.858401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.858399    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" containerName="revoke-keys"
Oct 30 13:33:05.858501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.858487    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0da0687-9a51-4371-9287-d56f620ac584" containerName="payload-processing"
Oct 30 13:33:05.858501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.858498    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c637ab26-9d39-4ab8-b1a6-32e0280e5eb4" containerName="revoke-keys"
Oct 30 13:33:05.861594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.861577    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:05.884184 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.884156    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"]
Oct 30 13:33:05.887246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.887220    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-143cf2\""
Oct 30 13:33:05.918733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:05.918700    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-143cf2-dockercfg-6l94s\""
Oct 30 13:33:06.090795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.090576    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"]
Oct 30 13:33:06.114708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.098482    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.120176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.115995    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"]
Oct 30 13:33:06.122686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.120795    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmkdc\" (UniqueName: \"kubernetes.io/projected/89611c42-963c-4c3d-bbb0-f0898cd76e92-kube-api-access-cmkdc\") pod \"payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") " pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.122915 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.122475    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"]
Oct 30 13:33:06.123097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.123074    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89611c42-963c-4c3d-bbb0-f0898cd76e92-plugins-config-volume\") pod \"payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") " pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.123243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.123225    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0481860-8e92-4d65-9cf9-49275b6183b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:06.123425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.123405    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bkrm\" (UniqueName: \"kubernetes.io/projected/f0481860-8e92-4d65-9cf9-49275b6183b3-kube-api-access-2bkrm\") pod \"payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:06.126160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.126137    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-418f95ff-k8cwq"]
Oct 30 13:33:06.224929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.224895    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmkdc\" (UniqueName: \"kubernetes.io/projected/89611c42-963c-4c3d-bbb0-f0898cd76e92-kube-api-access-cmkdc\") pod \"payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") " pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.225124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.225019    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89611c42-963c-4c3d-bbb0-f0898cd76e92-plugins-config-volume\") pod \"payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") " pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.225124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.225053    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0481860-8e92-4d65-9cf9-49275b6183b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:06.225124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.225098    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bkrm\" (UniqueName: \"kubernetes.io/projected/f0481860-8e92-4d65-9cf9-49275b6183b3-kube-api-access-2bkrm\") pod \"payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:06.225874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.225855    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89611c42-963c-4c3d-bbb0-f0898cd76e92-plugins-config-volume\") pod \"payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") " pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.226037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.226016    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0481860-8e92-4d65-9cf9-49275b6183b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:06.245497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.245463    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmkdc\" (UniqueName: \"kubernetes.io/projected/89611c42-963c-4c3d-bbb0-f0898cd76e92-kube-api-access-cmkdc\") pod \"payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") " pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.245497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.245487    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bkrm\" (UniqueName: \"kubernetes.io/projected/f0481860-8e92-4d65-9cf9-49275b6183b3-kube-api-access-2bkrm\") pod \"payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:06.437038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.436937    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:06.477899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.477868    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:06.743300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.743271    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"]
Oct 30 13:33:06.743429 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:33:06.743392    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89611c42_963c_4c3d_bbb0_f0898cd76e92.slice/crio-057da5e265d71e7db11bf133fe2090a3fef357fe79404e303a8d38e3feaa388a WatchSource:0}: Error finding container 057da5e265d71e7db11bf133fe2090a3fef357fe79404e303a8d38e3feaa388a: Status 404 returned error can't find the container with id 057da5e265d71e7db11bf133fe2090a3fef357fe79404e303a8d38e3feaa388a
Oct 30 13:33:06.797929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:06.797904    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"]
Oct 30 13:33:06.798660 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:33:06.798606    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0481860_8e92_4d65_9cf9_49275b6183b3.slice/crio-78ac4c4d0ba5c4943e59517e9b56b8a20a26ef7c633b1190552dd61f43e0a629 WatchSource:0}: Error finding container 78ac4c4d0ba5c4943e59517e9b56b8a20a26ef7c633b1190552dd61f43e0a629: Status 404 returned error can't find the container with id 78ac4c4d0ba5c4943e59517e9b56b8a20a26ef7c633b1190552dd61f43e0a629
Oct 30 13:33:07.639681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.639615    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm" event={"ID":"f0481860-8e92-4d65-9cf9-49275b6183b3","Type":"ContainerStarted","Data":"28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8"}
Oct 30 13:33:07.640156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.639860    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:07.640156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.639899    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm" event={"ID":"f0481860-8e92-4d65-9cf9-49275b6183b3","Type":"ContainerStarted","Data":"78ac4c4d0ba5c4943e59517e9b56b8a20a26ef7c633b1190552dd61f43e0a629"}
Oct 30 13:33:07.641442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.641416    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82" event={"ID":"89611c42-963c-4c3d-bbb0-f0898cd76e92","Type":"ContainerStarted","Data":"f366df1e6872c7046bf4898818b7718e8e73cabb867e211a4a88715700e6bece"}
Oct 30 13:33:07.641561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.641454    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82" event={"ID":"89611c42-963c-4c3d-bbb0-f0898cd76e92","Type":"ContainerStarted","Data":"057da5e265d71e7db11bf133fe2090a3fef357fe79404e303a8d38e3feaa388a"}
Oct 30 13:33:07.641561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.641473    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:07.676951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.676899    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm" podStartSLOduration=2.676885891 podStartE2EDuration="2.676885891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:07.675907223 +0000 UTC m=+1797.936145451" watchObservedRunningTime="2026-10-30 13:33:07.676885891 +0000 UTC m=+1797.937124180"
Oct 30 13:33:07.713550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.713502    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82" podStartSLOduration=2.71348907 podStartE2EDuration="2.71348907s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:33:07.713292573 +0000 UTC m=+1797.973530806" watchObservedRunningTime="2026-10-30 13:33:07.71348907 +0000 UTC m=+1797.973727303"
Oct 30 13:33:07.882362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:07.882318    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b4d15c-8109-4a74-ad03-e93cf6370992" path="/var/lib/kubelet/pods/25b4d15c-8109-4a74-ad03-e93cf6370992/volumes"
Oct 30 13:33:08.212504 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:33:08.212464    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:33:11.576494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:11.576460    4291 scope.go:122] "RemoveContainer" containerID="f691a0d033ecf81b887439bb8f5be521d9ea1de2ee049cfda3cdfac54715a1cc"
Oct 30 13:33:11.586580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:11.586548    4291 scope.go:122] "RemoveContainer" containerID="32f16672cbc08533ecbb3866fd8ab61fba5fc576a05a59bb4872eda2eb089987"
Oct 30 13:33:11.596297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:11.596276    4291 scope.go:122] "RemoveContainer" containerID="74bb4c4c6c67d6ee851c73f3cee36e9ee2ec2e4bbe6f8c92df5fc3312c894211"
Oct 30 13:33:11.605962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:11.605940    4291 scope.go:122] "RemoveContainer" containerID="6b12f76f41e126d7acae2d0e418fb3308af0c9ac1bc61c9d900ebb6a65e3d542"
Oct 30 13:33:11.615673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:11.615620    4291 scope.go:122] "RemoveContainer" containerID="655bebedf687f6dd37c237542966746ffad102302d8e5943bec2a889eb5d9a43"
Oct 30 13:33:11.625111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:11.625090    4291 scope.go:122] "RemoveContainer" containerID="6ac156cf3411a10fb182e105e7a9fe30f3079e090201214fb62742d24240c8c8"
Oct 30 13:33:16.033986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:16.033951    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"]
Oct 30 13:33:16.043291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:16.043257    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-273ad47d-84m2s"]
Oct 30 13:33:17.880795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:17.880759    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6b51a7-9805-4092-adab-1eb477500eb8" path="/var/lib/kubelet/pods/be6b51a7-9805-4092-adab-1eb477500eb8/volumes"
Oct 30 13:33:18.652001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:18.651961    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:33:18.652204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:18.652043    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:33:20.044237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:20.044206    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"]
Oct 30 13:33:20.055485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:20.055460    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6213b77f-p2ntd"]
Oct 30 13:33:21.880603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:21.880562    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7052ec8a-cdd4-496f-b2c6-d29275e1053e" path="/var/lib/kubelet/pods/7052ec8a-cdd4-496f-b2c6-d29275e1053e/volumes"
Oct 30 13:33:23.772799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:23.772763    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:33:33.034149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:33.034105    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"]
Oct 30 13:33:33.045763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:33.045722    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-471184b7-692h7"]
Oct 30 13:33:33.878901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:33.878869    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d975de3a-7140-4aaf-8d85-3fa69b46540d" path="/var/lib/kubelet/pods/d975de3a-7140-4aaf-8d85-3fa69b46540d/volumes"
Oct 30 13:33:35.035228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:35.035194    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"]
Oct 30 13:33:35.046744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:35.046711    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c7284524-pmd2g"]
Oct 30 13:33:35.880745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:35.880707    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e6452f-65c5-4d47-beb9-b821e19eeeae" path="/var/lib/kubelet/pods/14e6452f-65c5-4d47-beb9-b821e19eeeae/volumes"
Oct 30 13:33:38.213814 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:33:38.213772    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:33:43.031767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:43.031724    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"]
Oct 30 13:33:43.043293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:43.043261    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df7a53b6-dkl6c"]
Oct 30 13:33:43.879821 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:43.879787    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f" path="/var/lib/kubelet/pods/53cdf8b7-d83b-4bc0-a435-d31d9ae05f9f/volumes"
Oct 30 13:33:47.049148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:47.049101    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"]
Oct 30 13:33:47.060040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:47.059992    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1f64bfe5-966l8"]
Oct 30 13:33:47.879840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:47.879807    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0f3f304-7f78-425c-861a-e725febcc82c" path="/var/lib/kubelet/pods/d0f3f304-7f78-425c-861a-e725febcc82c/volumes"
Oct 30 13:33:54.042783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:54.042696    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"]
Oct 30 13:33:54.054532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:54.054499    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504d892-8rrt9"]
Oct 30 13:33:55.880453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:33:55.880413    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1f02c58-ea03-4c86-8a61-50b8bbe2e962" path="/var/lib/kubelet/pods/c1f02c58-ea03-4c86-8a61-50b8bbe2e962/volumes"
Oct 30 13:34:01.033420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:01.033386    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"]
Oct 30 13:34:01.046236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:01.046202    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fda6aa59-xncwx"]
Oct 30 13:34:01.884230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:01.884186    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3226023-6848-4f5d-bde3-933e0ff935b7" path="/var/lib/kubelet/pods/d3226023-6848-4f5d-bde3-933e0ff935b7/volumes"
Oct 30 13:34:02.295942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.295840    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"]
Oct 30 13:34:02.301942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.301915    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.316318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.316289    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"]
Oct 30 13:34:02.324193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.324143    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-143cf2\"/\"openshift-service-ca.crt\""
Oct 30 13:34:02.324379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.324194    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-143cf2\"/\"kube-root-ca.crt\""
Oct 30 13:34:02.327884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.327857    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-143cf2\"/\"default-dockercfg-c572j\""
Oct 30 13:34:02.328014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.327955    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-143cf2\"/\"explicit-model-143cf2-kserve-self-signed-certs\""
Oct 30 13:34:02.381957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.381918    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-dshm\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.382151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.381988    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ct9g\" (UniqueName: \"kubernetes.io/projected/5283c66f-cc55-406f-a19d-e853aa0feb6b-kube-api-access-2ct9g\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.382151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.382130    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5283c66f-cc55-406f-a19d-e853aa0feb6b-tls-certs\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.382246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.382233    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-model-cache\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.382281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.382259    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-tmp-dir\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.382325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.382295    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-home\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.483629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.483588    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-dshm\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.483629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.483662    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ct9g\" (UniqueName: \"kubernetes.io/projected/5283c66f-cc55-406f-a19d-e853aa0feb6b-kube-api-access-2ct9g\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.483937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.483693    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5283c66f-cc55-406f-a19d-e853aa0feb6b-tls-certs\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.483937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.483746    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-model-cache\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.483937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.483770    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-tmp-dir\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.483937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.483835    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-home\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.484422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.484397    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-tmp-dir\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.484517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.484403    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-home\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.484517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.484484    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-model-cache\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.486747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.486716    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-dshm\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.486945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.486924    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5283c66f-cc55-406f-a19d-e853aa0feb6b-tls-certs\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.504149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.504115    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ct9g\" (UniqueName: \"kubernetes.io/projected/5283c66f-cc55-406f-a19d-e853aa0feb6b-kube-api-access-2ct9g\") pod \"explicit-model-143cf2-kserve-85c8946cd5-dtscq\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") " pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.619939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.619900    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:02.993892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:02.993864    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"]
Oct 30 13:34:02.996067 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:34:02.996037    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5283c66f_cc55_406f_a19d_e853aa0feb6b.slice/crio-733c67654d69db394d82fe792102b62c6ab2cbe5eaa7495e0891bf70e144e253 WatchSource:0}: Error finding container 733c67654d69db394d82fe792102b62c6ab2cbe5eaa7495e0891bf70e144e253: Status 404 returned error can't find the container with id 733c67654d69db394d82fe792102b62c6ab2cbe5eaa7495e0891bf70e144e253
Oct 30 13:34:03.232903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:03.232866    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq" event={"ID":"5283c66f-cc55-406f-a19d-e853aa0feb6b","Type":"ContainerStarted","Data":"c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e"}
Oct 30 13:34:03.233093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:03.232913    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq" event={"ID":"5283c66f-cc55-406f-a19d-e853aa0feb6b","Type":"ContainerStarted","Data":"733c67654d69db394d82fe792102b62c6ab2cbe5eaa7495e0891bf70e144e253"}
Oct 30 13:34:03.279670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:03.279538    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq" podStartSLOduration=1.279514931 podStartE2EDuration="1.279514931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:03.275243673 +0000 UTC m=+1853.535481901" watchObservedRunningTime="2026-10-30 13:34:03.279514931 +0000 UTC m=+1853.539753168"
Oct 30 13:34:03.620955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:03.620921    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:08.212571 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:34:08.212529    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:34:09.032620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:09.032582    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"]
Oct 30 13:34:09.043960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:09.043929    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-00b53492-8c2sm"]
Oct 30 13:34:09.880990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:09.880952    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dee515e5-6ec0-499f-866b-86bdadeeb41f" path="/var/lib/kubelet/pods/dee515e5-6ec0-499f-866b-86bdadeeb41f/volumes"
Oct 30 13:34:11.032069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.032033    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"]
Oct 30 13:34:11.043833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.043799    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81c628e3-8scgn"]
Oct 30 13:34:11.686187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.686150    4291 scope.go:122] "RemoveContainer" containerID="6f0615fc60150431eaf88932384e21ed8fba3f62d78f9f4928b86ed6763551a8"
Oct 30 13:34:11.696346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.696319    4291 scope.go:122] "RemoveContainer" containerID="3bb80961b4d8f729dd6378e70594fbe84d6274ee37deaa93bada8c27dc9a3eb1"
Oct 30 13:34:11.706234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.706212    4291 scope.go:122] "RemoveContainer" containerID="a474256b66012db897ca927531b95a100caba5ba9c9cb8bb463c8e37a9f284be"
Oct 30 13:34:11.716035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.716014    4291 scope.go:122] "RemoveContainer" containerID="8edd27205c1edbf08ca8ff4f75ba8fb5f8676c1d59149039dc0515df53efb652"
Oct 30 13:34:11.726107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.725735    4291 scope.go:122] "RemoveContainer" containerID="81c73958288f75e5fa45054cd6c7d6a34fb6641445f39afd9c4ed9ee5be4a9fc"
Oct 30 13:34:11.735298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.735276    4291 scope.go:122] "RemoveContainer" containerID="14c5dfa24b3594516241b0cc2bb3ff68e30be925466d549a114a299b2f9c3255"
Oct 30 13:34:11.744810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.744788    4291 scope.go:122] "RemoveContainer" containerID="2f7175c36f64d03f3435f365c48984cc7dee006b12b3dd22254eb9549a65d5cf"
Oct 30 13:34:11.756503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.756479    4291 scope.go:122] "RemoveContainer" containerID="9d34cbe0286e3974c653d4dc012fb355e08c43af5881ffddf9a2750b0756f862"
Oct 30 13:34:11.766980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.766959    4291 scope.go:122] "RemoveContainer" containerID="cfe014a874c2612b84f9ddd7d576d01b656d4e79392d37d352bcd97360487b49"
Oct 30 13:34:11.880995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:11.880960    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4895a2f-5c45-44a6-a54f-e7741ce13bc4" path="/var/lib/kubelet/pods/a4895a2f-5c45-44a6-a54f-e7741ce13bc4/volumes"
Oct 30 13:34:12.621272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:12.621233    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:12.641211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:12.641182    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:13.316456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:13.316428    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:15.905205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:15.905172    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:34:20.072235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.072199    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"]
Oct 30 13:34:20.077226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.077202    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.091956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.091927    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:34:20.092406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.092387    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"]
Oct 30 13:34:20.100761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.100728    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:34:20.100996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.100974    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:34:20.103944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.103921    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:34:20.151381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.151343    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f063c777-7088-4e83-98e4-31f969253e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.151542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.151385    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qns5j\" (UniqueName: \"kubernetes.io/projected/f063c777-7088-4e83-98e4-31f969253e46-kube-api-access-qns5j\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.151598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.151588    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f063c777-7088-4e83-98e4-31f969253e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.252887 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.252849    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f063c777-7088-4e83-98e4-31f969253e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.252887 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.252889    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qns5j\" (UniqueName: \"kubernetes.io/projected/f063c777-7088-4e83-98e4-31f969253e46-kube-api-access-qns5j\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.253145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.253042    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f063c777-7088-4e83-98e4-31f969253e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.253727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.253703    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f063c777-7088-4e83-98e4-31f969253e46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.255320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.255299    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f063c777-7088-4e83-98e4-31f969253e46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.273495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.273459    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qns5j\" (UniqueName: \"kubernetes.io/projected/f063c777-7088-4e83-98e4-31f969253e46-kube-api-access-qns5j\") pod \"maas-api-revoke-sub-47161c45-fnrlg\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.391598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.391561    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:20.700480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:20.700447    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"]
Oct 30 13:34:20.701792 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:34:20.701765    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf063c777_7088_4e83_98e4_31f969253e46.slice/crio-b988d8f994c6243cd3b20085d62ffec33b8da9b0dd98fa9eb068b277161c94c4 WatchSource:0}: Error finding container b988d8f994c6243cd3b20085d62ffec33b8da9b0dd98fa9eb068b277161c94c4: Status 404 returned error can't find the container with id b988d8f994c6243cd3b20085d62ffec33b8da9b0dd98fa9eb068b277161c94c4
Oct 30 13:34:21.361353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:21.361307    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg" event={"ID":"f063c777-7088-4e83-98e4-31f969253e46","Type":"ContainerStarted","Data":"28b8a3f045c9ff1d2af4c8390cb83c324f592840fc7a8b7a55e9f0e922640e24"}
Oct 30 13:34:21.361781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:21.361375    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg" event={"ID":"f063c777-7088-4e83-98e4-31f969253e46","Type":"ContainerStarted","Data":"b988d8f994c6243cd3b20085d62ffec33b8da9b0dd98fa9eb068b277161c94c4"}
Oct 30 13:34:21.401086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:21.401022    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg" podStartSLOduration=1.401006554 podStartE2EDuration="1.401006554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:21.39824166 +0000 UTC m=+1871.658479894" watchObservedRunningTime="2026-10-30 13:34:21.401006554 +0000 UTC m=+1871.661244803"
Oct 30 13:34:22.370092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:22.370054    4291 generic.go:356] "Generic (PLEG): container finished" podID="f063c777-7088-4e83-98e4-31f969253e46" containerID="28b8a3f045c9ff1d2af4c8390cb83c324f592840fc7a8b7a55e9f0e922640e24" exitCode=0
Oct 30 13:34:22.370522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:22.370110    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg" event={"ID":"f063c777-7088-4e83-98e4-31f969253e46","Type":"ContainerDied","Data":"28b8a3f045c9ff1d2af4c8390cb83c324f592840fc7a8b7a55e9f0e922640e24"}
Oct 30 13:34:23.532767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.532733    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:23.601819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.601782    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f063c777-7088-4e83-98e4-31f969253e46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f063c777-7088-4e83-98e4-31f969253e46-maas-api-openshift-ca\") pod \"f063c777-7088-4e83-98e4-31f969253e46\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") "
Oct 30 13:34:23.602004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.601846    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f063c777-7088-4e83-98e4-31f969253e46-kube-api-access-qns5j\" (UniqueName: \"kubernetes.io/projected/f063c777-7088-4e83-98e4-31f969253e46-kube-api-access-qns5j\") pod \"f063c777-7088-4e83-98e4-31f969253e46\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") "
Oct 30 13:34:23.602004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.601899    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f063c777-7088-4e83-98e4-31f969253e46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f063c777-7088-4e83-98e4-31f969253e46-maas-api-xks-ca\") pod \"f063c777-7088-4e83-98e4-31f969253e46\" (UID: \"f063c777-7088-4e83-98e4-31f969253e46\") "
Oct 30 13:34:23.602391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.602358    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f063c777-7088-4e83-98e4-31f969253e46-maas-api-openshift-ca" pod "f063c777-7088-4e83-98e4-31f969253e46" (UID: "f063c777-7088-4e83-98e4-31f969253e46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:23.602527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.602483    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f063c777-7088-4e83-98e4-31f969253e46-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:23.604625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.604599    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f063c777-7088-4e83-98e4-31f969253e46-maas-api-xks-ca" pod "f063c777-7088-4e83-98e4-31f969253e46" (UID: "f063c777-7088-4e83-98e4-31f969253e46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:23.604741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.604664    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f063c777-7088-4e83-98e4-31f969253e46-kube-api-access-qns5j" pod "f063c777-7088-4e83-98e4-31f969253e46" (UID: "f063c777-7088-4e83-98e4-31f969253e46"). InnerVolumeSpecName "kube-api-access-qns5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:23.704083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.703989    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qns5j\" (UniqueName: \"kubernetes.io/projected/f063c777-7088-4e83-98e4-31f969253e46-kube-api-access-qns5j\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:23.704083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:23.704024    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f063c777-7088-4e83-98e4-31f969253e46-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:24.383996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:24.383969    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"
Oct 30 13:34:24.384203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:24.384007    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg" event={"ID":"f063c777-7088-4e83-98e4-31f969253e46","Type":"ContainerDied","Data":"b988d8f994c6243cd3b20085d62ffec33b8da9b0dd98fa9eb068b277161c94c4"}
Oct 30 13:34:24.384203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:24.384060    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b988d8f994c6243cd3b20085d62ffec33b8da9b0dd98fa9eb068b277161c94c4"
Oct 30 13:34:26.590626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.590596    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"]
Oct 30 13:34:26.964706 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.964668    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"]
Oct 30 13:34:26.965224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.965211    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f063c777-7088-4e83-98e4-31f969253e46" containerName="revoke-keys"
Oct 30 13:34:26.965271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.965226    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f063c777-7088-4e83-98e4-31f969253e46" containerName="revoke-keys"
Oct 30 13:34:26.965353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.965344    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f063c777-7088-4e83-98e4-31f969253e46" containerName="revoke-keys"
Oct 30 13:34:26.968962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.968941    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:26.985365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.985335    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:34:26.985954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.985934    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:34:26.986096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.986016    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:34:26.986170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.986106    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:34:26.987816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:26.987789    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"]
Oct 30 13:34:27.057982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.057935    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwd2l\" (UniqueName: \"kubernetes.io/projected/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-kube-api-access-nwd2l\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.058204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.058099    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.058204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.058155    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.159206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.159169    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.159424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.159405    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwd2l\" (UniqueName: \"kubernetes.io/projected/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-kube-api-access-nwd2l\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.159512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.159496    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.160146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.160113    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.161589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.161568    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.180867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.180838    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwd2l\" (UniqueName: \"kubernetes.io/projected/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-kube-api-access-nwd2l\") pod \"maas-api-revoke-keys-e2e-explicit-143cf2-kll4s\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.285535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.285444    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:27.600988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:27.600951    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"]
Oct 30 13:34:27.601461 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:34:27.601414    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e3df6d8_e9b8_43e2_a0de_e34735d01be9.slice/crio-d13944acb45bc2406887d13aab12a9c410862494831e6b9ff6600e3d93aea7a3 WatchSource:0}: Error finding container d13944acb45bc2406887d13aab12a9c410862494831e6b9ff6600e3d93aea7a3: Status 404 returned error can't find the container with id d13944acb45bc2406887d13aab12a9c410862494831e6b9ff6600e3d93aea7a3
Oct 30 13:34:28.417252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:28.417208    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s" event={"ID":"9e3df6d8-e9b8-43e2-a0de-e34735d01be9","Type":"ContainerStarted","Data":"3e05b00790031ba12b29caa2a5c803c400c3eef9f5b7b0a465b1ae4c870629aa"}
Oct 30 13:34:28.417440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:28.417279    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s" event={"ID":"9e3df6d8-e9b8-43e2-a0de-e34735d01be9","Type":"ContainerStarted","Data":"d13944acb45bc2406887d13aab12a9c410862494831e6b9ff6600e3d93aea7a3"}
Oct 30 13:34:28.457434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:28.457374    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s" podStartSLOduration=2.457355217 podStartE2EDuration="2.457355217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:34:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:34:28.457289021 +0000 UTC m=+1878.717527256" watchObservedRunningTime="2026-10-30 13:34:28.457355217 +0000 UTC m=+1878.717593452"
Oct 30 13:34:29.425817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:29.425785    4291 generic.go:356] "Generic (PLEG): container finished" podID="9e3df6d8-e9b8-43e2-a0de-e34735d01be9" containerID="3e05b00790031ba12b29caa2a5c803c400c3eef9f5b7b0a465b1ae4c870629aa" exitCode=0
Oct 30 13:34:29.426239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:29.425871    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s" event={"ID":"9e3df6d8-e9b8-43e2-a0de-e34735d01be9","Type":"ContainerDied","Data":"3e05b00790031ba12b29caa2a5c803c400c3eef9f5b7b0a465b1ae4c870629aa"}
Oct 30 13:34:30.487465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.487430    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"]
Oct 30 13:34:30.491532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.491509    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.514917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.514883    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"]
Oct 30 13:34:30.609140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.609107    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgqww\" (UniqueName: \"kubernetes.io/projected/6f64a4c7-144d-426e-adce-e158ffac8c40-kube-api-access-cgqww\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.609330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.609178    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.609330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.609308    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.621950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.621924    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:30.710281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.710178    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-xks-ca\") pod \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") "
Oct 30 13:34:30.710281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.710224    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-openshift-ca\") pod \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") "
Oct 30 13:34:30.710539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.710289    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-kube-api-access-nwd2l\" (UniqueName: \"kubernetes.io/projected/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-kube-api-access-nwd2l\") pod \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\" (UID: \"9e3df6d8-e9b8-43e2-a0de-e34735d01be9\") "
Oct 30 13:34:30.710539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.710500    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.710674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.710583    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.710674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.710623    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-openshift-ca" pod "9e3df6d8-e9b8-43e2-a0de-e34735d01be9" (UID: "9e3df6d8-e9b8-43e2-a0de-e34735d01be9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:30.711043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.711013    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgqww\" (UniqueName: \"kubernetes.io/projected/6f64a4c7-144d-426e-adce-e158ffac8c40-kube-api-access-cgqww\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.711185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.711171    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:30.711427 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.711401    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.712847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.712811    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-xks-ca" pod "9e3df6d8-e9b8-43e2-a0de-e34735d01be9" (UID: "9e3df6d8-e9b8-43e2-a0de-e34735d01be9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:30.712847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.712836    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-kube-api-access-nwd2l" pod "9e3df6d8-e9b8-43e2-a0de-e34735d01be9" (UID: "9e3df6d8-e9b8-43e2-a0de-e34735d01be9"). InnerVolumeSpecName "kube-api-access-nwd2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:30.713193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.713168    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.731978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.731946    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgqww\" (UniqueName: \"kubernetes.io/projected/6f64a4c7-144d-426e-adce-e158ffac8c40-kube-api-access-cgqww\") pod \"maas-api-revoke-sub-50cb5d3b-zbpwh\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:30.812089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.812054    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:30.812089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.812084    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwd2l\" (UniqueName: \"kubernetes.io/projected/9e3df6d8-e9b8-43e2-a0de-e34735d01be9-kube-api-access-nwd2l\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:30.818503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:30.818484    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:31.123951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:31.123925    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"]
Oct 30 13:34:31.124772 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:34:31.124741    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f64a4c7_144d_426e_adce_e158ffac8c40.slice/crio-4969b6669fd5b1c2dfea7a7efe4a1db0d2031a78d9b5566e500b6f2a80110299 WatchSource:0}: Error finding container 4969b6669fd5b1c2dfea7a7efe4a1db0d2031a78d9b5566e500b6f2a80110299: Status 404 returned error can't find the container with id 4969b6669fd5b1c2dfea7a7efe4a1db0d2031a78d9b5566e500b6f2a80110299
Oct 30 13:34:31.442064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:31.441971    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"
Oct 30 13:34:31.442064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:31.441997    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s" event={"ID":"9e3df6d8-e9b8-43e2-a0de-e34735d01be9","Type":"ContainerDied","Data":"d13944acb45bc2406887d13aab12a9c410862494831e6b9ff6600e3d93aea7a3"}
Oct 30 13:34:31.442064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:31.442054    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d13944acb45bc2406887d13aab12a9c410862494831e6b9ff6600e3d93aea7a3"
Oct 30 13:34:31.443578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:31.443553    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh" event={"ID":"6f64a4c7-144d-426e-adce-e158ffac8c40","Type":"ContainerStarted","Data":"4969b6669fd5b1c2dfea7a7efe4a1db0d2031a78d9b5566e500b6f2a80110299"}
Oct 30 13:34:32.452450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:32.452411    4291 generic.go:356] "Generic (PLEG): container finished" podID="6f64a4c7-144d-426e-adce-e158ffac8c40" containerID="2f20cd45ad032c4aa37fed0109f2e42d31e3f0ad0aa7bb9b45d40c9299120067" exitCode=0
Oct 30 13:34:32.452871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:32.452496    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh" event={"ID":"6f64a4c7-144d-426e-adce-e158ffac8c40","Type":"ContainerDied","Data":"2f20cd45ad032c4aa37fed0109f2e42d31e3f0ad0aa7bb9b45d40c9299120067"}
Oct 30 13:34:33.621272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.621245    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:33.752850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.752770    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-openshift-ca\") pod \"6f64a4c7-144d-426e-adce-e158ffac8c40\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") "
Oct 30 13:34:33.753032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.752892    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-xks-ca\") pod \"6f64a4c7-144d-426e-adce-e158ffac8c40\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") "
Oct 30 13:34:33.753130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.753108    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-openshift-ca" pod "6f64a4c7-144d-426e-adce-e158ffac8c40" (UID: "6f64a4c7-144d-426e-adce-e158ffac8c40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:33.753184 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.753144    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f64a4c7-144d-426e-adce-e158ffac8c40-kube-api-access-cgqww\" (UniqueName: \"kubernetes.io/projected/6f64a4c7-144d-426e-adce-e158ffac8c40-kube-api-access-cgqww\") pod \"6f64a4c7-144d-426e-adce-e158ffac8c40\" (UID: \"6f64a4c7-144d-426e-adce-e158ffac8c40\") "
Oct 30 13:34:33.753614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.753595    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:33.755108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.755086    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-xks-ca" pod "6f64a4c7-144d-426e-adce-e158ffac8c40" (UID: "6f64a4c7-144d-426e-adce-e158ffac8c40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:33.755200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.755110    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f64a4c7-144d-426e-adce-e158ffac8c40-kube-api-access-cgqww" pod "6f64a4c7-144d-426e-adce-e158ffac8c40" (UID: "6f64a4c7-144d-426e-adce-e158ffac8c40"). InnerVolumeSpecName "kube-api-access-cgqww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:33.854596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.854561    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f64a4c7-144d-426e-adce-e158ffac8c40-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:33.854788 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:33.854608    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgqww\" (UniqueName: \"kubernetes.io/projected/6f64a4c7-144d-426e-adce-e158ffac8c40-kube-api-access-cgqww\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:34.467425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:34.467387    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh" event={"ID":"6f64a4c7-144d-426e-adce-e158ffac8c40","Type":"ContainerDied","Data":"4969b6669fd5b1c2dfea7a7efe4a1db0d2031a78d9b5566e500b6f2a80110299"}
Oct 30 13:34:34.467604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:34.467446    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4969b6669fd5b1c2dfea7a7efe4a1db0d2031a78d9b5566e500b6f2a80110299"
Oct 30 13:34:34.467604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:34.467404    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"
Oct 30 13:34:37.358554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:37.358522    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"]
Oct 30 13:34:37.397378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:37.397347    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"]
Oct 30 13:34:38.212132 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:34:38.212095    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:34:41.617379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.617313    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq" podUID="5283c66f-cc55-406f-a19d-e853aa0feb6b" containerName="main" containerID="cri-o://c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e" gracePeriod=45
Oct 30 13:34:41.912704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.912675    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:41.968574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.968541    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5283c66f-cc55-406f-a19d-e853aa0feb6b-kube-api-access-2ct9g\" (UniqueName: \"kubernetes.io/projected/5283c66f-cc55-406f-a19d-e853aa0feb6b-kube-api-access-2ct9g\") pod \"5283c66f-cc55-406f-a19d-e853aa0feb6b\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") "
Oct 30 13:34:41.968763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.968603    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-dshm\") pod \"5283c66f-cc55-406f-a19d-e853aa0feb6b\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") "
Oct 30 13:34:41.968763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.968710    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-home\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-home\") pod \"5283c66f-cc55-406f-a19d-e853aa0feb6b\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") "
Oct 30 13:34:41.968909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.968845    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-tmp-dir\") pod \"5283c66f-cc55-406f-a19d-e853aa0feb6b\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") "
Oct 30 13:34:41.969080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.969023    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-home" pod "5283c66f-cc55-406f-a19d-e853aa0feb6b" (UID: "5283c66f-cc55-406f-a19d-e853aa0feb6b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:41.969212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.969105    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-tmp-dir" pod "5283c66f-cc55-406f-a19d-e853aa0feb6b" (UID: "5283c66f-cc55-406f-a19d-e853aa0feb6b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:41.969212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.969197    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5283c66f-cc55-406f-a19d-e853aa0feb6b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5283c66f-cc55-406f-a19d-e853aa0feb6b-tls-certs\") pod \"5283c66f-cc55-406f-a19d-e853aa0feb6b\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") "
Oct 30 13:34:41.969368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.969347    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-model-cache\") pod \"5283c66f-cc55-406f-a19d-e853aa0feb6b\" (UID: \"5283c66f-cc55-406f-a19d-e853aa0feb6b\") "
Oct 30 13:34:41.969613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.969584    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-model-cache" pod "5283c66f-cc55-406f-a19d-e853aa0feb6b" (UID: "5283c66f-cc55-406f-a19d-e853aa0feb6b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:41.970451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.970427    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:41.970572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.970560    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:41.970687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.970673    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:41.971316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.971288    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-dshm" pod "5283c66f-cc55-406f-a19d-e853aa0feb6b" (UID: "5283c66f-cc55-406f-a19d-e853aa0feb6b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:41.971509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.971488    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5283c66f-cc55-406f-a19d-e853aa0feb6b-kube-api-access-2ct9g" pod "5283c66f-cc55-406f-a19d-e853aa0feb6b" (UID: "5283c66f-cc55-406f-a19d-e853aa0feb6b"). InnerVolumeSpecName "kube-api-access-2ct9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:41.971773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:41.971754    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5283c66f-cc55-406f-a19d-e853aa0feb6b-tls-certs" pod "5283c66f-cc55-406f-a19d-e853aa0feb6b" (UID: "5283c66f-cc55-406f-a19d-e853aa0feb6b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:34:42.072173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.072135    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ct9g\" (UniqueName: \"kubernetes.io/projected/5283c66f-cc55-406f-a19d-e853aa0feb6b-kube-api-access-2ct9g\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:42.072173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.072167    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5283c66f-cc55-406f-a19d-e853aa0feb6b-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:42.072173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.072177    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5283c66f-cc55-406f-a19d-e853aa0feb6b-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:42.358843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.358782    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82" podUID="89611c42-963c-4c3d-bbb0-f0898cd76e92" containerName="payload-processing" containerID="cri-o://f366df1e6872c7046bf4898818b7718e8e73cabb867e211a4a88715700e6bece" gracePeriod=25
Oct 30 13:34:42.490508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.490448    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm" podUID="f0481860-8e92-4d65-9cf9-49275b6183b3" containerName="payload-pre-processing" containerID="cri-o://28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8" gracePeriod=25
Oct 30 13:34:42.525322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.525292    4291 generic.go:356] "Generic (PLEG): container finished" podID="89611c42-963c-4c3d-bbb0-f0898cd76e92" containerID="f366df1e6872c7046bf4898818b7718e8e73cabb867e211a4a88715700e6bece" exitCode=0
Oct 30 13:34:42.525490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.525359    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82" event={"ID":"89611c42-963c-4c3d-bbb0-f0898cd76e92","Type":"ContainerDied","Data":"f366df1e6872c7046bf4898818b7718e8e73cabb867e211a4a88715700e6bece"}
Oct 30 13:34:42.526982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.526940    4291 generic.go:356] "Generic (PLEG): container finished" podID="5283c66f-cc55-406f-a19d-e853aa0feb6b" containerID="c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e" exitCode=0
Oct 30 13:34:42.527117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.527002    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"
Oct 30 13:34:42.527117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.527021    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq" event={"ID":"5283c66f-cc55-406f-a19d-e853aa0feb6b","Type":"ContainerDied","Data":"c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e"}
Oct 30 13:34:42.527117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.527060    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq" event={"ID":"5283c66f-cc55-406f-a19d-e853aa0feb6b","Type":"ContainerDied","Data":"733c67654d69db394d82fe792102b62c6ab2cbe5eaa7495e0891bf70e144e253"}
Oct 30 13:34:42.527117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.527078    4291 scope.go:122] "RemoveContainer" containerID="c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e"
Oct 30 13:34:42.586748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.586712    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"]
Oct 30 13:34:42.605962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.605939    4291 scope.go:122] "RemoveContainer" containerID="c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e"
Oct 30 13:34:42.606467 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:34:42.606437    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e\": container with ID starting with c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e not found: ID does not exist" containerID="c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e"
Oct 30 13:34:42.606578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.606486    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e"} err="failed to get container status \"c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e\": rpc error: code = NotFound desc = could not find container \"c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e\": container with ID starting with c516955a3567cc03ba3ddea0324ecd429f2dd1a7310a42d60ef18914c648401e not found: ID does not exist"
Oct 30 13:34:42.620516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.620480    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-143cf2/explicit-model-143cf2-kserve-85c8946cd5-dtscq"]
Oct 30 13:34:42.658490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.658464    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:34:42.683111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.680898    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89611c42-963c-4c3d-bbb0-f0898cd76e92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89611c42-963c-4c3d-bbb0-f0898cd76e92-plugins-config-volume\") pod \"89611c42-963c-4c3d-bbb0-f0898cd76e92\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") "
Oct 30 13:34:42.683111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.681444    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89611c42-963c-4c3d-bbb0-f0898cd76e92-plugins-config-volume" pod "89611c42-963c-4c3d-bbb0-f0898cd76e92" (UID: "89611c42-963c-4c3d-bbb0-f0898cd76e92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:42.683111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.681781    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89611c42-963c-4c3d-bbb0-f0898cd76e92-kube-api-access-cmkdc\" (UniqueName: \"kubernetes.io/projected/89611c42-963c-4c3d-bbb0-f0898cd76e92-kube-api-access-cmkdc\") pod \"89611c42-963c-4c3d-bbb0-f0898cd76e92\" (UID: \"89611c42-963c-4c3d-bbb0-f0898cd76e92\") "
Oct 30 13:34:42.683111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.683093    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89611c42-963c-4c3d-bbb0-f0898cd76e92-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:42.689667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.685816    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89611c42-963c-4c3d-bbb0-f0898cd76e92-kube-api-access-cmkdc" pod "89611c42-963c-4c3d-bbb0-f0898cd76e92" (UID: "89611c42-963c-4c3d-bbb0-f0898cd76e92"). InnerVolumeSpecName "kube-api-access-cmkdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:42.785038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.784995    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cmkdc\" (UniqueName: \"kubernetes.io/projected/89611c42-963c-4c3d-bbb0-f0898cd76e92-kube-api-access-cmkdc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:42.790325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.790301    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:34:42.885774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.885690    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0481860-8e92-4d65-9cf9-49275b6183b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0481860-8e92-4d65-9cf9-49275b6183b3-plugins-config-volume\") pod \"f0481860-8e92-4d65-9cf9-49275b6183b3\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") "
Oct 30 13:34:42.885960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.885783    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0481860-8e92-4d65-9cf9-49275b6183b3-kube-api-access-2bkrm\" (UniqueName: \"kubernetes.io/projected/f0481860-8e92-4d65-9cf9-49275b6183b3-kube-api-access-2bkrm\") pod \"f0481860-8e92-4d65-9cf9-49275b6183b3\" (UID: \"f0481860-8e92-4d65-9cf9-49275b6183b3\") "
Oct 30 13:34:42.886067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.886044    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0481860-8e92-4d65-9cf9-49275b6183b3-plugins-config-volume" pod "f0481860-8e92-4d65-9cf9-49275b6183b3" (UID: "f0481860-8e92-4d65-9cf9-49275b6183b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:42.886367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.886345    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0481860-8e92-4d65-9cf9-49275b6183b3-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:42.889447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.889418    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0481860-8e92-4d65-9cf9-49275b6183b3-kube-api-access-2bkrm" pod "f0481860-8e92-4d65-9cf9-49275b6183b3" (UID: "f0481860-8e92-4d65-9cf9-49275b6183b3"). InnerVolumeSpecName "kube-api-access-2bkrm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:42.988160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:42.988114    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bkrm\" (UniqueName: \"kubernetes.io/projected/f0481860-8e92-4d65-9cf9-49275b6183b3-kube-api-access-2bkrm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:43.537411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.537373    4291 generic.go:356] "Generic (PLEG): container finished" podID="f0481860-8e92-4d65-9cf9-49275b6183b3" containerID="28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8" exitCode=0
Oct 30 13:34:43.537614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.537426    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"
Oct 30 13:34:43.537614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.537453    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm" event={"ID":"f0481860-8e92-4d65-9cf9-49275b6183b3","Type":"ContainerDied","Data":"28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8"}
Oct 30 13:34:43.537614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.537512    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm" event={"ID":"f0481860-8e92-4d65-9cf9-49275b6183b3","Type":"ContainerDied","Data":"78ac4c4d0ba5c4943e59517e9b56b8a20a26ef7c633b1190552dd61f43e0a629"}
Oct 30 13:34:43.537614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.537542    4291 scope.go:122] "RemoveContainer" containerID="28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8"
Oct 30 13:34:43.539129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.539109    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"
Oct 30 13:34:43.539215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.539107    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82" event={"ID":"89611c42-963c-4c3d-bbb0-f0898cd76e92","Type":"ContainerDied","Data":"057da5e265d71e7db11bf133fe2090a3fef357fe79404e303a8d38e3feaa388a"}
Oct 30 13:34:43.548551 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.548529    4291 scope.go:122] "RemoveContainer" containerID="28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8"
Oct 30 13:34:43.548897 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:34:43.548872    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8\": container with ID starting with 28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8 not found: ID does not exist" containerID="28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8"
Oct 30 13:34:43.548964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.548915    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8"} err="failed to get container status \"28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8\": rpc error: code = NotFound desc = could not find container \"28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8\": container with ID starting with 28961316c8e6b912ff223e31a15204db46a4afeb98d274e48cd4bb22dacb24b8 not found: ID does not exist"
Oct 30 13:34:43.548964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.548942    4291 scope.go:122] "RemoveContainer" containerID="f366df1e6872c7046bf4898818b7718e8e73cabb867e211a4a88715700e6bece"
Oct 30 13:34:43.590852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.590820    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"]
Oct 30 13:34:43.602345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.602315    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-143cf2-78ff8c77dd-k5r82"]
Oct 30 13:34:43.627012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.626980    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"]
Oct 30 13:34:43.632734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.632701    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-143cf2-ff68fb5ff-pwxlm"]
Oct 30 13:34:43.881261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.881220    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5283c66f-cc55-406f-a19d-e853aa0feb6b" path="/var/lib/kubelet/pods/5283c66f-cc55-406f-a19d-e853aa0feb6b/volumes"
Oct 30 13:34:43.881798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.881781    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89611c42-963c-4c3d-bbb0-f0898cd76e92" path="/var/lib/kubelet/pods/89611c42-963c-4c3d-bbb0-f0898cd76e92/volumes"
Oct 30 13:34:43.882278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:43.882263    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0481860-8e92-4d65-9cf9-49275b6183b3" path="/var/lib/kubelet/pods/f0481860-8e92-4d65-9cf9-49275b6183b3/volumes"
Oct 30 13:34:45.111063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:45.111025    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"]
Oct 30 13:34:45.122819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:45.122787    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-143cf2-kll4s"]
Oct 30 13:34:45.879627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:45.879592    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e3df6d8-e9b8-43e2-a0de-e34735d01be9" path="/var/lib/kubelet/pods/9e3df6d8-e9b8-43e2-a0de-e34735d01be9/volumes"
Oct 30 13:34:53.997941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:53.997897    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"]
Oct 30 13:34:53.998408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:53.998117    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy" containerID="cri-o://0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645" gracePeriod=30
Oct 30 13:34:59.267783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.267720    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:34:59.379823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.379792    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-token\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.379823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.379836    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-data\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.379857    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-credential-socket\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.379890    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-envoy\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.379936    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-kube-api-access-bw27l\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-kube-api-access-bw27l\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380154    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-data" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:59.380256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380158    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-podinfo\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380178    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-credential-socket" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:59.380408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380265    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istiod-ca-cert\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380298    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-socket\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380381    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-certs\") pod \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\" (UID: \"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d\") "
Oct 30 13:34:59.380585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380563    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-socket" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:59.380815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380696    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istiod-ca-cert" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:34:59.380902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.380783    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-certs" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:59.381126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.381104    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.381192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.381131    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.381192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.381146    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.381192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.381160    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.381192 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.381174    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.382160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.382132    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-token" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:59.382262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.382176    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-envoy" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:34:59.382262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.382237    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-podinfo" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:34:59.382369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.382352    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-kube-api-access-bw27l" pod "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" (UID: "02d3ae08-43a5-4453-b9ba-b1da9eb2d41d"). InnerVolumeSpecName "kube-api-access-bw27l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:34:59.482827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.482798    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.482827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.482824    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.482827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.482832    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.483071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.482839    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bw27l\" (UniqueName: \"kubernetes.io/projected/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d-kube-api-access-bw27l\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:34:59.715222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.715187    4291 generic.go:356] "Generic (PLEG): container finished" podID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerID="0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645" exitCode=0
Oct 30 13:34:59.715417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.715240    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"
Oct 30 13:34:59.715417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.715272    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" event={"ID":"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d","Type":"ContainerDied","Data":"0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645"}
Oct 30 13:34:59.715417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.715329    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54" event={"ID":"02d3ae08-43a5-4453-b9ba-b1da9eb2d41d","Type":"ContainerDied","Data":"c2e27ecd8f2d6251a0710534e842cc728bfd2c093ba71292d1b0c943f49d935a"}
Oct 30 13:34:59.715417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.715359    4291 scope.go:122] "RemoveContainer" containerID="0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645"
Oct 30 13:34:59.726619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.726599    4291 scope.go:122] "RemoveContainer" containerID="0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645"
Oct 30 13:34:59.726948 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:34:59.726925    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645\": container with ID starting with 0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645 not found: ID does not exist" containerID="0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645"
Oct 30 13:34:59.727030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.726957    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645"} err="failed to get container status \"0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645\": rpc error: code = NotFound desc = could not find container \"0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645\": container with ID starting with 0d9bd8e51c55c7fb2e83bf849a3110670ed4b96da43f020e67ba524d7e339645 not found: ID does not exist"
Oct 30 13:34:59.769589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.769558    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"]
Oct 30 13:34:59.782078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.782036    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2jt54"]
Oct 30 13:34:59.884456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:34:59.884416    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" path="/var/lib/kubelet/pods/02d3ae08-43a5-4453-b9ba-b1da9eb2d41d/volumes"
Oct 30 13:35:05.032842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:05.032808    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"]
Oct 30 13:35:05.044473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:05.044433    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-67fdb213-2k68g"]
Oct 30 13:35:05.878807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:05.878773    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1a3415e-33ca-4d86-819c-ea4eb26eb3f8" path="/var/lib/kubelet/pods/c1a3415e-33ca-4d86-819c-ea4eb26eb3f8/volumes"
Oct 30 13:35:07.578395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:07.578360    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:35:08.271439 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:35:08.271394    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:35:11.839679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.839628    4291 scope.go:122] "RemoveContainer" containerID="56abbcb8cc6fad5df183305e101e0483a8ceedfdaefcccfb63c99a8fe62dd21a"
Oct 30 13:35:11.849447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.849428    4291 scope.go:122] "RemoveContainer" containerID="6f8cb84ee78e23f83b288200f154b5c368dfe25579dcd5524493e4475479ce5a"
Oct 30 13:35:11.859180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.859156    4291 scope.go:122] "RemoveContainer" containerID="bc9c7847db49d92a29e1237bb0165262b55c03847c414efaa75f784ee458825d"
Oct 30 13:35:11.867587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.867568    4291 scope.go:122] "RemoveContainer" containerID="8bb28d0dc26797732c673de90c9768af82f28ba71b878db50e8134d70e685216"
Oct 30 13:35:11.877119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.877025    4291 scope.go:122] "RemoveContainer" containerID="f8e82d9a0a8aa8d42bf58099141637ac1fd06060cde55320160c3fa8c31956c8"
Oct 30 13:35:11.886373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.886356    4291 scope.go:122] "RemoveContainer" containerID="8e15e14f7131d6d48c5cf0e42d62dfda8eea4f0bd2337753bf99674f5280275b"
Oct 30 13:35:11.896359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.896328    4291 scope.go:122] "RemoveContainer" containerID="c21e08a1800f8ce80bec75a2a58edb8d250eb35905cd1f4e298398208a497f3c"
Oct 30 13:35:11.904809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.904792    4291 scope.go:122] "RemoveContainer" containerID="e2e525ec7293a1b24df994f0741d6f9967964d37ff1e92cab352bcba64d4dc10"
Oct 30 13:35:11.912963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.912934    4291 scope.go:122] "RemoveContainer" containerID="b7a4e73b9bbc4032b311f66422179cd020f099eb14590f72dc1154ddda09f703"
Oct 30 13:35:11.921399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.921380    4291 scope.go:122] "RemoveContainer" containerID="1ee0b072e2595479d05e019e617bda73405d71e4a5f09627f91660e1ab82f0d1"
Oct 30 13:35:11.929559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:11.929543    4291 scope.go:122] "RemoveContainer" containerID="65d319539b21ed5355e98355725c0ad71edadf259acc9dcfc1d1a63e6518ad27"
Oct 30 13:35:12.038951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.038912    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"]
Oct 30 13:35:12.039205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.039163    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy" containerID="cri-o://9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be" gracePeriod=30
Oct 30 13:35:12.194448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.194413    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"]
Oct 30 13:35:12.194991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.194973    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6f64a4c7-144d-426e-adce-e158ffac8c40" containerName="revoke-keys"
Oct 30 13:35:12.195064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.194993    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f64a4c7-144d-426e-adce-e158ffac8c40" containerName="revoke-keys"
Oct 30 13:35:12.195064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195021    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5283c66f-cc55-406f-a19d-e853aa0feb6b" containerName="main"
Oct 30 13:35:12.195064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195027    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5283c66f-cc55-406f-a19d-e853aa0feb6b" containerName="main"
Oct 30 13:35:12.195064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195035    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9e3df6d8-e9b8-43e2-a0de-e34735d01be9" containerName="revoke-keys"
Oct 30 13:35:12.195064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195040    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e3df6d8-e9b8-43e2-a0de-e34735d01be9" containerName="revoke-keys"
Oct 30 13:35:12.195064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195058    4291 cpu_manager.go:414] "containerMap: removing container" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy"
Oct 30 13:35:12.195064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195063    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195072    4291 cpu_manager.go:414] "containerMap: removing container" podUID="89611c42-963c-4c3d-bbb0-f0898cd76e92" containerName="payload-processing"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195077    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89611c42-963c-4c3d-bbb0-f0898cd76e92" containerName="payload-processing"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195088    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f0481860-8e92-4d65-9cf9-49275b6183b3" containerName="payload-pre-processing"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195093    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0481860-8e92-4d65-9cf9-49275b6183b3" containerName="payload-pre-processing"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195171    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="02d3ae08-43a5-4453-b9ba-b1da9eb2d41d" containerName="istio-proxy"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195180    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="89611c42-963c-4c3d-bbb0-f0898cd76e92" containerName="payload-processing"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195187    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0481860-8e92-4d65-9cf9-49275b6183b3" containerName="payload-pre-processing"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195194    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f64a4c7-144d-426e-adce-e158ffac8c40" containerName="revoke-keys"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195203    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5283c66f-cc55-406f-a19d-e853aa0feb6b" containerName="main"
Oct 30 13:35:12.195294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.195208    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e3df6d8-e9b8-43e2-a0de-e34735d01be9" containerName="revoke-keys"
Oct 30 13:35:12.200914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.200894    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.213806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.213784    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"]
Oct 30 13:35:12.214739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.214720    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:35:12.215140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.215120    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:35:12.215443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.215428    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:35:12.215522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.215493    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:35:12.293282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.293243    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzwdm\" (UniqueName: \"kubernetes.io/projected/1a6095a1-318c-4365-8625-1a639cdd268b-kube-api-access-zzwdm\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.293468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.293311    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.293556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.293532    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.334369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.334332    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"]
Oct 30 13:35:12.338265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.338242    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.355283 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.355257    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"]
Oct 30 13:35:12.395220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.395183    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzwdm\" (UniqueName: \"kubernetes.io/projected/1a6095a1-318c-4365-8625-1a639cdd268b-kube-api-access-zzwdm\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.395408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.395232    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.395408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.395395    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.396080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.396053    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.397757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.397735    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.413872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.413843    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzwdm\" (UniqueName: \"kubernetes.io/projected/1a6095a1-318c-4365-8625-1a639cdd268b-kube-api-access-zzwdm\") pod \"maas-api-revoke-sub-ac9725aa-bf78b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.496927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.496842    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.497086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.497043    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.497150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.497137    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxlq5\" (UniqueName: \"kubernetes.io/projected/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-kube-api-access-gxlq5\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.515614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.515577    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:12.598509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.598478    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.598708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.598630    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxlq5\" (UniqueName: \"kubernetes.io/projected/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-kube-api-access-gxlq5\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.598708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.598707    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.599238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.599206    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.601744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.601717    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.618300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.618272    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxlq5\" (UniqueName: \"kubernetes.io/projected/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-kube-api-access-gxlq5\") pod \"maas-api-revoke-sub-a5243cc8-5swhb\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.652148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.652106    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:12.832553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.832524    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"]
Oct 30 13:35:12.833753 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:35:12.833732    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6095a1_318c_4365_8625_1a639cdd268b.slice/crio-7ee129348f450fa01ba33a61da56ce22dbbc3764bda160073ab7173e4302850a WatchSource:0}: Error finding container 7ee129348f450fa01ba33a61da56ce22dbbc3764bda160073ab7173e4302850a: Status 404 returned error can't find the container with id 7ee129348f450fa01ba33a61da56ce22dbbc3764bda160073ab7173e4302850a
Oct 30 13:35:12.973458 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:35:12.973430    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc04780_ecf5_4600_9e56_5d6b0f5c42d6.slice/crio-662e3122e594a28d6679cd25e7aa4b495e1e7249474f97fde0bc341318b3b667 WatchSource:0}: Error finding container 662e3122e594a28d6679cd25e7aa4b495e1e7249474f97fde0bc341318b3b667: Status 404 returned error can't find the container with id 662e3122e594a28d6679cd25e7aa4b495e1e7249474f97fde0bc341318b3b667
Oct 30 13:35:12.974949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:12.974920    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"]
Oct 30 13:35:13.816023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:13.815985    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb" event={"ID":"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6","Type":"ContainerStarted","Data":"c0cc03468771aff165bf897edb9546fe047da49a3e7e42097c78c835a14abda9"}
Oct 30 13:35:13.816223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:13.816077    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb" event={"ID":"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6","Type":"ContainerStarted","Data":"662e3122e594a28d6679cd25e7aa4b495e1e7249474f97fde0bc341318b3b667"}
Oct 30 13:35:13.817567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:13.817538    4291 generic.go:356] "Generic (PLEG): container finished" podID="1a6095a1-318c-4365-8625-1a639cdd268b" containerID="5a1d814f023f60b107ce07d08505203fcd579f148b01442388bd50e6e0c5ae2a" exitCode=0
Oct 30 13:35:13.817727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:13.817588    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b" event={"ID":"1a6095a1-318c-4365-8625-1a639cdd268b","Type":"ContainerDied","Data":"5a1d814f023f60b107ce07d08505203fcd579f148b01442388bd50e6e0c5ae2a"}
Oct 30 13:35:13.817727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:13.817629    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b" event={"ID":"1a6095a1-318c-4365-8625-1a639cdd268b","Type":"ContainerStarted","Data":"7ee129348f450fa01ba33a61da56ce22dbbc3764bda160073ab7173e4302850a"}
Oct 30 13:35:13.867008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:13.866951    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb" podStartSLOduration=1.866938927 podStartE2EDuration="1.866938927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:13.853075048 +0000 UTC m=+1924.113313284" watchObservedRunningTime="2026-10-30 13:35:13.866938927 +0000 UTC m=+1924.127177187"
Oct 30 13:35:14.829459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:14.829423    4291 generic.go:356] "Generic (PLEG): container finished" podID="3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" containerID="c0cc03468771aff165bf897edb9546fe047da49a3e7e42097c78c835a14abda9" exitCode=0
Oct 30 13:35:14.830115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:14.830088    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb" event={"ID":"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6","Type":"ContainerDied","Data":"c0cc03468771aff165bf897edb9546fe047da49a3e7e42097c78c835a14abda9"}
Oct 30 13:35:14.996176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:14.996148    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:15.038293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.038262    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a6095a1-318c-4365-8625-1a639cdd268b-kube-api-access-zzwdm\" (UniqueName: \"kubernetes.io/projected/1a6095a1-318c-4365-8625-1a639cdd268b-kube-api-access-zzwdm\") pod \"1a6095a1-318c-4365-8625-1a639cdd268b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") "
Oct 30 13:35:15.038497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.038330    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-xks-ca\") pod \"1a6095a1-318c-4365-8625-1a639cdd268b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") "
Oct 30 13:35:15.038594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.038530    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-openshift-ca\") pod \"1a6095a1-318c-4365-8625-1a639cdd268b\" (UID: \"1a6095a1-318c-4365-8625-1a639cdd268b\") "
Oct 30 13:35:15.039266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.039237    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-openshift-ca" pod "1a6095a1-318c-4365-8625-1a639cdd268b" (UID: "1a6095a1-318c-4365-8625-1a639cdd268b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:15.042154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.041539    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a6095a1-318c-4365-8625-1a639cdd268b-kube-api-access-zzwdm" pod "1a6095a1-318c-4365-8625-1a639cdd268b" (UID: "1a6095a1-318c-4365-8625-1a639cdd268b"). InnerVolumeSpecName "kube-api-access-zzwdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:15.042154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.041883    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-xks-ca" pod "1a6095a1-318c-4365-8625-1a639cdd268b" (UID: "1a6095a1-318c-4365-8625-1a639cdd268b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:15.139992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.139957    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzwdm\" (UniqueName: \"kubernetes.io/projected/1a6095a1-318c-4365-8625-1a639cdd268b-kube-api-access-zzwdm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.139992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.139989    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.139992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.139998    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a6095a1-318c-4365-8625-1a639cdd268b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:15.837475 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.837443    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"
Oct 30 13:35:15.837907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.837477    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b" event={"ID":"1a6095a1-318c-4365-8625-1a639cdd268b","Type":"ContainerDied","Data":"7ee129348f450fa01ba33a61da56ce22dbbc3764bda160073ab7173e4302850a"}
Oct 30 13:35:15.837907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.837535    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ee129348f450fa01ba33a61da56ce22dbbc3764bda160073ab7173e4302850a"
Oct 30 13:35:15.999508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:15.999488    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:16.052421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.052391    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-openshift-ca\") pod \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") "
Oct 30 13:35:16.052579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.052494    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-xks-ca\") pod \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") "
Oct 30 13:35:16.052617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.052577    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-kube-api-access-gxlq5\" (UniqueName: \"kubernetes.io/projected/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-kube-api-access-gxlq5\") pod \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\" (UID: \"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6\") "
Oct 30 13:35:16.052975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.052835    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-openshift-ca" pod "3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" (UID: "3bc04780-ecf5-4600-9e56-5d6b0f5c42d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:16.053334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.053313    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:16.055378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.055340    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-xks-ca" pod "3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" (UID: "3bc04780-ecf5-4600-9e56-5d6b0f5c42d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:16.057274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.057251    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-kube-api-access-gxlq5" pod "3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" (UID: "3bc04780-ecf5-4600-9e56-5d6b0f5c42d6"). InnerVolumeSpecName "kube-api-access-gxlq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:16.154933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.154903    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:16.154933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.154932    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxlq5\" (UniqueName: \"kubernetes.io/projected/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6-kube-api-access-gxlq5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:16.845546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.845518    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"
Oct 30 13:35:16.845950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.845558    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb" event={"ID":"3bc04780-ecf5-4600-9e56-5d6b0f5c42d6","Type":"ContainerDied","Data":"662e3122e594a28d6679cd25e7aa4b495e1e7249474f97fde0bc341318b3b667"}
Oct 30 13:35:16.845950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:16.845607    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="662e3122e594a28d6679cd25e7aa4b495e1e7249474f97fde0bc341318b3b667"
Oct 30 13:35:17.338280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.338253    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:35:17.369767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.369741    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-token\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-token\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.369904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.369795    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-credential-socket\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.369904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.369842    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-certs\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.369904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.369863    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-envoy\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.370061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.369921    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-socket\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.370061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370025    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istiod-ca-cert\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.370176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370102    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-podinfo\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.370176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370136    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-kube-api-access-gqrv4\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-kube-api-access-gqrv4\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.370176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370146    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-certs" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:17.370176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370161    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-credential-socket" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:17.370387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370179    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-data\") pod \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\" (UID: \"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218\") "
Oct 30 13:35:17.370451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370377    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-socket" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:17.370577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370551    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-data" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:17.370677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370625    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istiod-ca-cert" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:17.370852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370828    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.370852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370852    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.371034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370865    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.371034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370879    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.371034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.370892    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.372200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.372174    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-token" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:17.372540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.372516    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-envoy" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:17.372747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.372721    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-podinfo" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:35:17.372747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.372727    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-kube-api-access-gqrv4" pod "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" (UID: "9e213319-f5d9-4cb6-9ca5-cb6a38a4d218"). InnerVolumeSpecName "kube-api-access-gqrv4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:17.472772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.472677    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.472772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.472713    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqrv4\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-kube-api-access-gqrv4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.472772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.472729    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.472772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.472743    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:17.853953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.853919    4291 generic.go:356] "Generic (PLEG): container finished" podID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerID="9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be" exitCode=0
Oct 30 13:35:17.854344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.853971    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"
Oct 30 13:35:17.854344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.854006    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" event={"ID":"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218","Type":"ContainerDied","Data":"9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be"}
Oct 30 13:35:17.854344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.854090    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4" event={"ID":"9e213319-f5d9-4cb6-9ca5-cb6a38a4d218","Type":"ContainerDied","Data":"1c649aa421c2c31f305602ef2aab79b14d6251b5bfb246619e6039fb08c297b2"}
Oct 30 13:35:17.854344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.854116    4291 scope.go:122] "RemoveContainer" containerID="9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be"
Oct 30 13:35:17.864399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.864379    4291 scope.go:122] "RemoveContainer" containerID="9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be"
Oct 30 13:35:17.864678 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:35:17.864662    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be\": container with ID starting with 9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be not found: ID does not exist" containerID="9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be"
Oct 30 13:35:17.864737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.864686    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be"} err="failed to get container status \"9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be\": rpc error: code = NotFound desc = could not find container \"9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be\": container with ID starting with 9b84179b256c1449683f8a98f7d4b6d2c8084e5af2b0e2ec594c98b7d8faa4be not found: ID does not exist"
Oct 30 13:35:17.916040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.916012    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"]
Oct 30 13:35:17.928101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:17.928075    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2nnw4"]
Oct 30 13:35:19.882841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:19.880412    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" path="/var/lib/kubelet/pods/9e213319-f5d9-4cb6-9ca5-cb6a38a4d218/volumes"
Oct 30 13:35:22.594925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.594895    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"]
Oct 30 13:35:22.595383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595370    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy"
Oct 30 13:35:22.595430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595384    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy"
Oct 30 13:35:22.595430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595400    4291 cpu_manager.go:414] "containerMap: removing container" podUID="3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" containerName="revoke-keys"
Oct 30 13:35:22.595430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595405    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" containerName="revoke-keys"
Oct 30 13:35:22.595430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595430    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1a6095a1-318c-4365-8625-1a639cdd268b" containerName="revoke-keys"
Oct 30 13:35:22.595570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595435    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a6095a1-318c-4365-8625-1a639cdd268b" containerName="revoke-keys"
Oct 30 13:35:22.595570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595501    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e213319-f5d9-4cb6-9ca5-cb6a38a4d218" containerName="istio-proxy"
Oct 30 13:35:22.595570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595513    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" containerName="revoke-keys"
Oct 30 13:35:22.595570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.595524    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a6095a1-318c-4365-8625-1a639cdd268b" containerName="revoke-keys"
Oct 30 13:35:22.600185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.600170    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.615272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.615249    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"]
Oct 30 13:35:22.623183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.623142    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:35:22.623183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.623155    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:35:22.623437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.623240    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:35:22.625797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.625768    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:35:22.635515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.635485    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.635723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.635700    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.635843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.635821    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmq79\" (UniqueName: \"kubernetes.io/projected/c024db84-2b2c-4e26-9638-56e764b23a48-kube-api-access-gmq79\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.702827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.702793    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"]
Oct 30 13:35:22.706983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.706960    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.723541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.723517    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"]
Oct 30 13:35:22.737185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.737159    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.737318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.737238    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmq79\" (UniqueName: \"kubernetes.io/projected/c024db84-2b2c-4e26-9638-56e764b23a48-kube-api-access-gmq79\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.737318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.737282    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txdwh\" (UniqueName: \"kubernetes.io/projected/92ce11af-deba-4546-99cd-c319034911c9-kube-api-access-txdwh\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.737444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.737415    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce11af-deba-4546-99cd-c319034911c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.737548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.737512    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.737621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.737604    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce11af-deba-4546-99cd-c319034911c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.738342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.738313    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.739664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.739647    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.756510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.756482    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmq79\" (UniqueName: \"kubernetes.io/projected/c024db84-2b2c-4e26-9638-56e764b23a48-kube-api-access-gmq79\") pod \"maas-api-revoke-sub-95e51fcb-cbdzh\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:22.838436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.838376    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txdwh\" (UniqueName: \"kubernetes.io/projected/92ce11af-deba-4546-99cd-c319034911c9-kube-api-access-txdwh\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.838534 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.838480    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce11af-deba-4546-99cd-c319034911c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.838594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.838550    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce11af-deba-4546-99cd-c319034911c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.839182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.839159    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce11af-deba-4546-99cd-c319034911c9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.840773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.840756    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce11af-deba-4546-99cd-c319034911c9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.857689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.857624    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txdwh\" (UniqueName: \"kubernetes.io/projected/92ce11af-deba-4546-99cd-c319034911c9-kube-api-access-txdwh\") pod \"maas-api-revoke-sub-5ca2b64f-wnmcr\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:22.913338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:22.913314    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:23.021059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.021025    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:23.214630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.214605    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"]
Oct 30 13:35:23.216752 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:35:23.216721    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc024db84_2b2c_4e26_9638_56e764b23a48.slice/crio-2c535110218f70b9487b0a86ccd40ff43cd0062061abd2c88401e5bbb731348b WatchSource:0}: Error finding container 2c535110218f70b9487b0a86ccd40ff43cd0062061abd2c88401e5bbb731348b: Status 404 returned error can't find the container with id 2c535110218f70b9487b0a86ccd40ff43cd0062061abd2c88401e5bbb731348b
Oct 30 13:35:23.332438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.332418    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"]
Oct 30 13:35:23.333334 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:35:23.333311    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ce11af_deba_4546_99cd_c319034911c9.slice/crio-e877aa7d77fae328cba3d3759df14398d0f0da62ad714c2424808e56430392a4 WatchSource:0}: Error finding container e877aa7d77fae328cba3d3759df14398d0f0da62ad714c2424808e56430392a4: Status 404 returned error can't find the container with id e877aa7d77fae328cba3d3759df14398d0f0da62ad714c2424808e56430392a4
Oct 30 13:35:23.898154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.898120    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr" event={"ID":"92ce11af-deba-4546-99cd-c319034911c9","Type":"ContainerStarted","Data":"39a945d8adbbbd4f461b64b4bedef5303f0d500c38618dc7342cb0da34911994"}
Oct 30 13:35:23.898667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.898167    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr" event={"ID":"92ce11af-deba-4546-99cd-c319034911c9","Type":"ContainerStarted","Data":"e877aa7d77fae328cba3d3759df14398d0f0da62ad714c2424808e56430392a4"}
Oct 30 13:35:23.899776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.899750    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh" event={"ID":"c024db84-2b2c-4e26-9638-56e764b23a48","Type":"ContainerStarted","Data":"0a6ed0ec41bb0286034993dfc5aa022bcb0b1a0f3014fbb7910f145882a94542"}
Oct 30 13:35:23.899875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.899793    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh" event={"ID":"c024db84-2b2c-4e26-9638-56e764b23a48","Type":"ContainerStarted","Data":"2c535110218f70b9487b0a86ccd40ff43cd0062061abd2c88401e5bbb731348b"}
Oct 30 13:35:23.934875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.934834    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr" podStartSLOduration=1.93482229 podStartE2EDuration="1.93482229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:23.934334588 +0000 UTC m=+1934.194572821" watchObservedRunningTime="2026-10-30 13:35:23.93482229 +0000 UTC m=+1934.195060523"
Oct 30 13:35:23.972488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:23.972440    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh" podStartSLOduration=1.972427793 podStartE2EDuration="1.972427793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:23.970209232 +0000 UTC m=+1934.230447467" watchObservedRunningTime="2026-10-30 13:35:23.972427793 +0000 UTC m=+1934.232666027"
Oct 30 13:35:24.907779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:24.907744    4291 generic.go:356] "Generic (PLEG): container finished" podID="c024db84-2b2c-4e26-9638-56e764b23a48" containerID="0a6ed0ec41bb0286034993dfc5aa022bcb0b1a0f3014fbb7910f145882a94542" exitCode=0
Oct 30 13:35:24.908280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:24.907841    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh" event={"ID":"c024db84-2b2c-4e26-9638-56e764b23a48","Type":"ContainerDied","Data":"0a6ed0ec41bb0286034993dfc5aa022bcb0b1a0f3014fbb7910f145882a94542"}
Oct 30 13:35:24.909459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:24.909435    4291 generic.go:356] "Generic (PLEG): container finished" podID="92ce11af-deba-4546-99cd-c319034911c9" containerID="39a945d8adbbbd4f461b64b4bedef5303f0d500c38618dc7342cb0da34911994" exitCode=0
Oct 30 13:35:24.909589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:24.909483    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr" event={"ID":"92ce11af-deba-4546-99cd-c319034911c9","Type":"ContainerDied","Data":"39a945d8adbbbd4f461b64b4bedef5303f0d500c38618dc7342cb0da34911994"}
Oct 30 13:35:26.124451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.124421    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:26.180440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.180364    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:26.187397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.187363    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c024db84-2b2c-4e26-9638-56e764b23a48-kube-api-access-gmq79\" (UniqueName: \"kubernetes.io/projected/c024db84-2b2c-4e26-9638-56e764b23a48-kube-api-access-gmq79\") pod \"c024db84-2b2c-4e26-9638-56e764b23a48\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") "
Oct 30 13:35:26.187745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.187724    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-openshift-ca\") pod \"c024db84-2b2c-4e26-9638-56e764b23a48\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") "
Oct 30 13:35:26.187864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.187771    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-xks-ca\") pod \"c024db84-2b2c-4e26-9638-56e764b23a48\" (UID: \"c024db84-2b2c-4e26-9638-56e764b23a48\") "
Oct 30 13:35:26.188307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.188101    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-openshift-ca" pod "c024db84-2b2c-4e26-9638-56e764b23a48" (UID: "c024db84-2b2c-4e26-9638-56e764b23a48"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:26.188402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.188327    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:26.190287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.190257    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c024db84-2b2c-4e26-9638-56e764b23a48-kube-api-access-gmq79" pod "c024db84-2b2c-4e26-9638-56e764b23a48" (UID: "c024db84-2b2c-4e26-9638-56e764b23a48"). InnerVolumeSpecName "kube-api-access-gmq79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:26.190480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.190454    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-xks-ca" pod "c024db84-2b2c-4e26-9638-56e764b23a48" (UID: "c024db84-2b2c-4e26-9638-56e764b23a48"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:26.289677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.289624    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92ce11af-deba-4546-99cd-c319034911c9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce11af-deba-4546-99cd-c319034911c9-maas-api-xks-ca\") pod \"92ce11af-deba-4546-99cd-c319034911c9\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") "
Oct 30 13:35:26.289840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.289693    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ce11af-deba-4546-99cd-c319034911c9-kube-api-access-txdwh\" (UniqueName: \"kubernetes.io/projected/92ce11af-deba-4546-99cd-c319034911c9-kube-api-access-txdwh\") pod \"92ce11af-deba-4546-99cd-c319034911c9\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") "
Oct 30 13:35:26.289840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.289826    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ce11af-deba-4546-99cd-c319034911c9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce11af-deba-4546-99cd-c319034911c9-maas-api-openshift-ca\") pod \"92ce11af-deba-4546-99cd-c319034911c9\" (UID: \"92ce11af-deba-4546-99cd-c319034911c9\") "
Oct 30 13:35:26.290201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.290176    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ce11af-deba-4546-99cd-c319034911c9-maas-api-openshift-ca" pod "92ce11af-deba-4546-99cd-c319034911c9" (UID: "92ce11af-deba-4546-99cd-c319034911c9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:26.290293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.290259    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c024db84-2b2c-4e26-9638-56e764b23a48-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:26.290293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.290278    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmq79\" (UniqueName: \"kubernetes.io/projected/c024db84-2b2c-4e26-9638-56e764b23a48-kube-api-access-gmq79\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:26.291928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.291901    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92ce11af-deba-4546-99cd-c319034911c9-maas-api-xks-ca" pod "92ce11af-deba-4546-99cd-c319034911c9" (UID: "92ce11af-deba-4546-99cd-c319034911c9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:26.292322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.292301    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ce11af-deba-4546-99cd-c319034911c9-kube-api-access-txdwh" pod "92ce11af-deba-4546-99cd-c319034911c9" (UID: "92ce11af-deba-4546-99cd-c319034911c9"). InnerVolumeSpecName "kube-api-access-txdwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:26.392672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.392450    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92ce11af-deba-4546-99cd-c319034911c9-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:26.392672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.392490    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92ce11af-deba-4546-99cd-c319034911c9-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:26.392672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.392505    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txdwh\" (UniqueName: \"kubernetes.io/projected/92ce11af-deba-4546-99cd-c319034911c9-kube-api-access-txdwh\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:26.925761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.925718    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"
Oct 30 13:35:26.925761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.925742    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr" event={"ID":"92ce11af-deba-4546-99cd-c319034911c9","Type":"ContainerDied","Data":"e877aa7d77fae328cba3d3759df14398d0f0da62ad714c2424808e56430392a4"}
Oct 30 13:35:26.926040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.925804    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e877aa7d77fae328cba3d3759df14398d0f0da62ad714c2424808e56430392a4"
Oct 30 13:35:26.927350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.927328    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"
Oct 30 13:35:26.927453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.927399    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh" event={"ID":"c024db84-2b2c-4e26-9638-56e764b23a48","Type":"ContainerDied","Data":"2c535110218f70b9487b0a86ccd40ff43cd0062061abd2c88401e5bbb731348b"}
Oct 30 13:35:26.927453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:26.927436    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c535110218f70b9487b0a86ccd40ff43cd0062061abd2c88401e5bbb731348b"
Oct 30 13:35:34.646075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:34.645989    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"]
Oct 30 13:35:38.215889 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:35:38.215846    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:35:44.056790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:44.056755    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"]
Oct 30 13:35:44.057252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:44.056975    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" containerID="cri-o://6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd" gracePeriod=30
Oct 30 13:35:45.249407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.249360    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"]
Oct 30 13:35:45.249961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.249930    4291 cpu_manager.go:414] "containerMap: removing container" podUID="92ce11af-deba-4546-99cd-c319034911c9" containerName="revoke-keys"
Oct 30 13:35:45.249961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.249945    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ce11af-deba-4546-99cd-c319034911c9" containerName="revoke-keys"
Oct 30 13:35:45.250092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.249983    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c024db84-2b2c-4e26-9638-56e764b23a48" containerName="revoke-keys"
Oct 30 13:35:45.250092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.249988    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c024db84-2b2c-4e26-9638-56e764b23a48" containerName="revoke-keys"
Oct 30 13:35:45.250092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.250077    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c024db84-2b2c-4e26-9638-56e764b23a48" containerName="revoke-keys"
Oct 30 13:35:45.250092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.250086    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ce11af-deba-4546-99cd-c319034911c9" containerName="revoke-keys"
Oct 30 13:35:45.253575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.253551    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.275294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.275259    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"]
Oct 30 13:35:45.279961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.279936    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-bd2fed-dockercfg-6wtw9\""
Oct 30 13:35:45.349982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.349926    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-bd2fed\""
Oct 30 13:35:45.400547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.400502    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5gk5\" (UniqueName: \"kubernetes.io/projected/ed7440e2-c465-496c-8172-161a37301ab9-kube-api-access-v5gk5\") pod \"payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.400840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.400603    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7440e2-c465-496c-8172-161a37301ab9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.509984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.508907    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5gk5\" (UniqueName: \"kubernetes.io/projected/ed7440e2-c465-496c-8172-161a37301ab9-kube-api-access-v5gk5\") pod \"payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.509984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.509013    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7440e2-c465-496c-8172-161a37301ab9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.510233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.510206    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7440e2-c465-496c-8172-161a37301ab9-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.528930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.528899    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5gk5\" (UniqueName: \"kubernetes.io/projected/ed7440e2-c465-496c-8172-161a37301ab9-kube-api-access-v5gk5\") pod \"payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.574121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.574084    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:45.889059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:45.889024    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"]
Oct 30 13:35:45.890744 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:35:45.890712    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded7440e2_c465_496c_8172_161a37301ab9.slice/crio-bb8dc626a4aea418a9af099521cf5fca5cd5bb61b1f6b5b56de5ae9048d1cd86 WatchSource:0}: Error finding container bb8dc626a4aea418a9af099521cf5fca5cd5bb61b1f6b5b56de5ae9048d1cd86: Status 404 returned error can't find the container with id bb8dc626a4aea418a9af099521cf5fca5cd5bb61b1f6b5b56de5ae9048d1cd86
Oct 30 13:35:46.064966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:46.064923    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" event={"ID":"ed7440e2-c465-496c-8172-161a37301ab9","Type":"ContainerStarted","Data":"5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93"}
Oct 30 13:35:46.065139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:46.065026    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" event={"ID":"ed7440e2-c465-496c-8172-161a37301ab9","Type":"ContainerStarted","Data":"bb8dc626a4aea418a9af099521cf5fca5cd5bb61b1f6b5b56de5ae9048d1cd86"}
Oct 30 13:35:46.065139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:46.065056    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:35:46.104845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:46.104788    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" podStartSLOduration=1.104773433 podStartE2EDuration="1.104773433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:35:46.102549762 +0000 UTC m=+1956.362787996" watchObservedRunningTime="2026-10-30 13:35:46.104773433 +0000 UTC m=+1956.365011706"
Oct 30 13:35:48.182958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:48.182923    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-xb65f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:35:48.183398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:48.182979    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:35:49.362249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.362220    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:35:49.417838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.417802    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-kube-api-access-8dsgw\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-kube-api-access-8dsgw\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.417923    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-token\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-token\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.417993    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-credential-socket\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418110 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418034    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-podinfo\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418120    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-certs\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418179    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-socket\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418239    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istiod-ca-cert\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418274    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-data\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418369    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-envoy\") pod \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\" (UID: \"88650bd5-e6e2-496b-a2c8-f58b9624eafb\") "
Oct 30 13:35:49.418858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418567    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-socket" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:49.418858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.418613    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-credential-socket" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:49.419098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.419023    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.419098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.419048    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.419284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.419258    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-data" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:49.419357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.419295    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istiod-ca-cert" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:35:49.419518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.419491    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-certs" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:49.423083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.423050    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-podinfo" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:35:49.423221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.423125    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-envoy" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:49.423221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.423143    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-kube-api-access-8dsgw" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "kube-api-access-8dsgw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:49.423221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.423161    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-token" pod "88650bd5-e6e2-496b-a2c8-f58b9624eafb" (UID: "88650bd5-e6e2-496b-a2c8-f58b9624eafb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:49.520610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.520576    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.520610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.520606    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.520610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.520616    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.520874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.520626    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.520874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.520647    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.520874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.520657    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dsgw\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-kube-api-access-8dsgw\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.520874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.520666    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/88650bd5-e6e2-496b-a2c8-f58b9624eafb-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:49.669764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.669723    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r" podUID="85b0d393-40b4-45d4-bb8f-2b13d049c78a" containerName="main" containerID="cri-o://137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01" gracePeriod=45
Oct 30 13:35:49.780873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.780835    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r" podUID="85b0d393-40b4-45d4-bb8f-2b13d049c78a" containerName="main" probeResult="failure" output="Get \"https://10.133.0.175:8000/ready\": dial tcp 10.133.0.175:8000: connect: connection refused"
Oct 30 13:35:49.940425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:49.940395    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.026909    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-dshm\") pod \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") "
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.026978    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-model-cache\") pod \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") "
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.027054    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-home\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-home\") pod \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") "
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.027075    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85b0d393-40b4-45d4-bb8f-2b13d049c78a-kube-api-access-k972k\" (UniqueName: \"kubernetes.io/projected/85b0d393-40b4-45d4-bb8f-2b13d049c78a-kube-api-access-k972k\") pod \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") "
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.027137    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tls-certs\" (UniqueName: \"kubernetes.io/secret/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tls-certs\") pod \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") "
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.027248    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tmp-dir\") pod \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\" (UID: \"85b0d393-40b4-45d4-bb8f-2b13d049c78a\") "
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.027294    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-home" pod "85b0d393-40b4-45d4-bb8f-2b13d049c78a" (UID: "85b0d393-40b4-45d4-bb8f-2b13d049c78a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.027403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.027308    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-model-cache" pod "85b0d393-40b4-45d4-bb8f-2b13d049c78a" (UID: "85b0d393-40b4-45d4-bb8f-2b13d049c78a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.029067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.028922    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.029067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.029067    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.029731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.029697    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tmp-dir" pod "85b0d393-40b4-45d4-bb8f-2b13d049c78a" (UID: "85b0d393-40b4-45d4-bb8f-2b13d049c78a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.030170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.030146    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-dshm" pod "85b0d393-40b4-45d4-bb8f-2b13d049c78a" (UID: "85b0d393-40b4-45d4-bb8f-2b13d049c78a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:35:50.033878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.031919    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tls-certs" pod "85b0d393-40b4-45d4-bb8f-2b13d049c78a" (UID: "85b0d393-40b4-45d4-bb8f-2b13d049c78a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:35:50.033878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.032356    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85b0d393-40b4-45d4-bb8f-2b13d049c78a-kube-api-access-k972k" pod "85b0d393-40b4-45d4-bb8f-2b13d049c78a" (UID: "85b0d393-40b4-45d4-bb8f-2b13d049c78a"). InnerVolumeSpecName "kube-api-access-k972k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:35:50.126063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.126027    4291 generic.go:356] "Generic (PLEG): container finished" podID="85b0d393-40b4-45d4-bb8f-2b13d049c78a" containerID="137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01" exitCode=0
Oct 30 13:35:50.126279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.126088    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"
Oct 30 13:35:50.126279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.126104    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r" event={"ID":"85b0d393-40b4-45d4-bb8f-2b13d049c78a","Type":"ContainerDied","Data":"137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01"}
Oct 30 13:35:50.126279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.126158    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r" event={"ID":"85b0d393-40b4-45d4-bb8f-2b13d049c78a","Type":"ContainerDied","Data":"1ccfdabe4971d8dd64712e2d872574473658f2b636f8889f55e0f65dcc952733"}
Oct 30 13:35:50.126279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.126182    4291 scope.go:122] "RemoveContainer" containerID="137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01"
Oct 30 13:35:50.128393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.128357    4291 generic.go:356] "Generic (PLEG): container finished" podID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerID="6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd" exitCode=0
Oct 30 13:35:50.128512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.128446    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"
Oct 30 13:35:50.128579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.128443    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" event={"ID":"88650bd5-e6e2-496b-a2c8-f58b9624eafb","Type":"ContainerDied","Data":"6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd"}
Oct 30 13:35:50.128666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.128615    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f" event={"ID":"88650bd5-e6e2-496b-a2c8-f58b9624eafb","Type":"ContainerDied","Data":"2a22e7f094062a22f742f6b43d217647bc759abc5e22287e36fbe075f7789c1f"}
Oct 30 13:35:50.130130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.130101    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.130130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.130134    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.130325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.130148    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/85b0d393-40b4-45d4-bb8f-2b13d049c78a-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.130325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.130162    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k972k\" (UniqueName: \"kubernetes.io/projected/85b0d393-40b4-45d4-bb8f-2b13d049c78a-kube-api-access-k972k\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:35:50.176101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.175006    4291 scope.go:122] "RemoveContainer" containerID="137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01"
Oct 30 13:35:50.176101 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:35:50.175521    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01\": container with ID starting with 137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01 not found: ID does not exist" containerID="137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01"
Oct 30 13:35:50.176101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.175558    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01"} err="failed to get container status \"137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01\": rpc error: code = NotFound desc = could not find container \"137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01\": container with ID starting with 137757c43d4b1355698c316532a2c9fe6af8cc583acc6abfa3b99bfefb138e01 not found: ID does not exist"
Oct 30 13:35:50.176101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.175583    4291 scope.go:122] "RemoveContainer" containerID="6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd"
Oct 30 13:35:50.187231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.186956    4291 scope.go:122] "RemoveContainer" containerID="6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd"
Oct 30 13:35:50.187644 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:35:50.187589    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd\": container with ID starting with 6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd not found: ID does not exist" containerID="6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd"
Oct 30 13:35:50.187770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.187670    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd"} err="failed to get container status \"6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd\": rpc error: code = NotFound desc = could not find container \"6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd\": container with ID starting with 6ec8331430e1312244f158fa5502487f9139048b8c516e647108de150dc200dd not found: ID does not exist"
Oct 30 13:35:50.193306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.193277    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"]
Oct 30 13:35:50.204808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.204772    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-xb65f"]
Oct 30 13:35:50.232064 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.232028    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"]
Oct 30 13:35:50.234998 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:50.234970    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-167965/auth-test-model-167965-kserve-55c895896f-bh67r"]
Oct 30 13:35:51.882851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:51.882814    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85b0d393-40b4-45d4-bb8f-2b13d049c78a" path="/var/lib/kubelet/pods/85b0d393-40b4-45d4-bb8f-2b13d049c78a/volumes"
Oct 30 13:35:51.883576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:51.883554    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" path="/var/lib/kubelet/pods/88650bd5-e6e2-496b-a2c8-f58b9624eafb/volumes"
Oct 30 13:35:57.099762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:35:57.099722    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 13:36:01.762660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.762056    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"]
Oct 30 13:36:01.763203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.763175    4291 cpu_manager.go:414] "containerMap: removing container" podUID="85b0d393-40b4-45d4-bb8f-2b13d049c78a" containerName="main"
Oct 30 13:36:01.763203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.763199    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85b0d393-40b4-45d4-bb8f-2b13d049c78a" containerName="main"
Oct 30 13:36:01.763350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.763213    4291 cpu_manager.go:414] "containerMap: removing container" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy"
Oct 30 13:36:01.763350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.763221    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy"
Oct 30 13:36:01.763350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.763346    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="85b0d393-40b4-45d4-bb8f-2b13d049c78a" containerName="main"
Oct 30 13:36:01.763510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.763356    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="88650bd5-e6e2-496b-a2c8-f58b9624eafb" containerName="istio-proxy"
Oct 30 13:36:01.774658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.772791    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.802661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.795117    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-167965-openshift-default-dockercfg-rfp6j\""
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808239    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808355    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808385    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808450    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808498    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808544    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808568    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808604    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.814679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.808768    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w98qf\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-kube-api-access-w98qf\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.847657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.832548    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"]
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.911708    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.911753    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.911797    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.911947    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w98qf\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-kube-api-access-w98qf\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.912018    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.912119    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.912150    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.912225    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-credential-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.912235    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.912325    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.912667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.912559    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-socket\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.914487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.914462    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-data\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.914924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.914880    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-certs\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.915678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.915658    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istiod-ca-cert\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.919806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.919783    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-envoy\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.923572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.923541    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-podinfo\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.937025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.936992    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-token\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:01.941545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:01.941509    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w98qf\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-kube-api-access-w98qf\") pod \"e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") " pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:02.125154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:02.125115    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:02.460337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:02.460310    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"]
Oct 30 13:36:02.461559 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:02.461534    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf386f4e9_bf64_44f0_ba8d_d02e5966cb9b.slice/crio-25bb0a701c660c3d36c12b9f5414cce9d23f46c4d47bcfefd1a6e1cb2fb3fff3 WatchSource:0}: Error finding container 25bb0a701c660c3d36c12b9f5414cce9d23f46c4d47bcfefd1a6e1cb2fb3fff3: Status 404 returned error can't find the container with id 25bb0a701c660c3d36c12b9f5414cce9d23f46c4d47bcfefd1a6e1cb2fb3fff3
Oct 30 13:36:02.464418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:02.464385    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:02.464487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:02.464462    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:02.464529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:02.464493    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:03.220681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.220565    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" event={"ID":"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b","Type":"ContainerStarted","Data":"3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f"}
Oct 30 13:36:03.220681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.220620    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" event={"ID":"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b","Type":"ContainerStarted","Data":"25bb0a701c660c3d36c12b9f5414cce9d23f46c4d47bcfefd1a6e1cb2fb3fff3"}
Oct 30 13:36:03.465661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.464661    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podStartSLOduration=2.464618765 podStartE2EDuration="2.464618765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:03.290227543 +0000 UTC m=+1973.550465778" watchObservedRunningTime="2026-10-30 13:36:03.464618765 +0000 UTC m=+1973.724857000"
Oct 30 13:36:03.470661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.467174    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"]
Oct 30 13:36:03.480472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.480385    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.488304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.488265    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"]
Oct 30 13:36:03.504378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.504355    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-51dc9a\""
Oct 30 13:36:03.504554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.504389    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-51dc9a-dockercfg-vp64q\""
Oct 30 13:36:03.519397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.519360    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"]
Oct 30 13:36:03.524253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.524231    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:03.533068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.533043    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7j68\" (UniqueName: \"kubernetes.io/projected/dd484118-cad8-4221-b60c-bb9aa540641d-kube-api-access-l7j68\") pod \"payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.533193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.533082    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd484118-cad8-4221-b60c-bb9aa540641d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.622661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.618737    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"]
Oct 30 13:36:03.661656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.648580    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7j68\" (UniqueName: \"kubernetes.io/projected/dd484118-cad8-4221-b60c-bb9aa540641d-kube-api-access-l7j68\") pod \"payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.661656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.648622    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd484118-cad8-4221-b60c-bb9aa540641d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.661656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.648790    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb91e01a-4783-41b2-9306-8ff29c0787f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:03.661656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.648923    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpvcq\" (UniqueName: \"kubernetes.io/projected/cb91e01a-4783-41b2-9306-8ff29c0787f0-kube-api-access-gpvcq\") pod \"payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:03.661656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.650527    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd484118-cad8-4221-b60c-bb9aa540641d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.725657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.722448    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7j68\" (UniqueName: \"kubernetes.io/projected/dd484118-cad8-4221-b60c-bb9aa540641d-kube-api-access-l7j68\") pod \"payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.753713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.753386    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpvcq\" (UniqueName: \"kubernetes.io/projected/cb91e01a-4783-41b2-9306-8ff29c0787f0-kube-api-access-gpvcq\") pod \"payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:03.753713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.753587    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb91e01a-4783-41b2-9306-8ff29c0787f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:03.773660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.754601    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb91e01a-4783-41b2-9306-8ff29c0787f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:03.812658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.812136    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:03.843699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.832490    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpvcq\" (UniqueName: \"kubernetes.io/projected/cb91e01a-4783-41b2-9306-8ff29c0787f0-kube-api-access-gpvcq\") pod \"payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:03.843699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:03.843464    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:04.125840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:04.125806    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:04.127498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:04.127466    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:36:04.127625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:04.127522    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:36:04.180469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:04.180440    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"]
Oct 30 13:36:04.181504 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:04.181475    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb91e01a_4783_41b2_9306_8ff29c0787f0.slice/crio-7cde95daf30631b900a7ca5602b23ebf60a4d35177a73c0b974cb7fe4841cd84 WatchSource:0}: Error finding container 7cde95daf30631b900a7ca5602b23ebf60a4d35177a73c0b974cb7fe4841cd84: Status 404 returned error can't find the container with id 7cde95daf30631b900a7ca5602b23ebf60a4d35177a73c0b974cb7fe4841cd84
Oct 30 13:36:04.201484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:04.201456    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"]
Oct 30 13:36:04.203373 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:04.203345    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd484118_cad8_4221_b60c_bb9aa540641d.slice/crio-4a376ec5788fb8b859e0f554e48ce230201b54c5dde9901623a3163de2334f93 WatchSource:0}: Error finding container 4a376ec5788fb8b859e0f554e48ce230201b54c5dde9901623a3163de2334f93: Status 404 returned error can't find the container with id 4a376ec5788fb8b859e0f554e48ce230201b54c5dde9901623a3163de2334f93
Oct 30 13:36:04.233845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:04.233813    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" event={"ID":"cb91e01a-4783-41b2-9306-8ff29c0787f0","Type":"ContainerStarted","Data":"7cde95daf30631b900a7ca5602b23ebf60a4d35177a73c0b974cb7fe4841cd84"}
Oct 30 13:36:04.235734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:04.235699    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" event={"ID":"dd484118-cad8-4221-b60c-bb9aa540641d","Type":"ContainerStarted","Data":"4a376ec5788fb8b859e0f554e48ce230201b54c5dde9901623a3163de2334f93"}
Oct 30 13:36:05.125424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.125391    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:36:05.125653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.125455    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:36:05.243452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.243415    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" event={"ID":"cb91e01a-4783-41b2-9306-8ff29c0787f0","Type":"ContainerStarted","Data":"90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1"}
Oct 30 13:36:05.243962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.243476    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:05.245186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.245161    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" event={"ID":"dd484118-cad8-4221-b60c-bb9aa540641d","Type":"ContainerStarted","Data":"1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4"}
Oct 30 13:36:05.245299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.245227    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:05.287926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.287873    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" podStartSLOduration=2.287858455 podStartE2EDuration="2.287858455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:05.285563013 +0000 UTC m=+1975.545801246" watchObservedRunningTime="2026-10-30 13:36:05.287858455 +0000 UTC m=+1975.548096689"
Oct 30 13:36:05.328481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:05.328429    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" podStartSLOduration=2.328415172 podStartE2EDuration="2.328415172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:05.323921897 +0000 UTC m=+1975.584160133" watchObservedRunningTime="2026-10-30 13:36:05.328415172 +0000 UTC m=+1975.588653405"
Oct 30 13:36:06.125897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:06.125857    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:36:06.126183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:06.125931    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:36:07.125768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:07.125736    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:36:07.126220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:07.125793    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:36:08.125431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:08.125399    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 13:36:08.125622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:08.125465    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 13:36:08.212139 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:36:08.212100    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:36:09.130470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:09.130442    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:09.130753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:09.130733    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:09.131350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:09.131333    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 13:36:11.761862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:11.761179    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:36:11.963842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:11.963799    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"]
Oct 30 13:36:11.964084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:11.964061    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" containerID="cri-o://514a23ba4d8550e36f82910d7d3a5a86fe6fad422db32bfb4d19a69889ea574b" gracePeriod=30
Oct 30 13:36:11.975468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:11.975439    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"]
Oct 30 13:36:11.975699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:11.975673    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy" containerID="cri-o://a20baaf687e71cf58e1e1710972e314913f7d7e0b79704215983041dfe20063c" gracePeriod=30
Oct 30 13:36:11.978818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:11.978760    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"]
Oct 30 13:36:11.979471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:11.979431    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy" containerID="cri-o://458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15" gracePeriod=30
Oct 30 13:36:12.018479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:12.018407    4291 scope.go:122] "RemoveContainer" containerID="13f99fa7c06ca3f9e9738497ab7d5bfc7a096eee4f0a87d2c9ce143100317bab"
Oct 30 13:36:12.028538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:12.028516    4291 scope.go:122] "RemoveContainer" containerID="cb405afc95009a42b6f7195b97780150bc081ea370efa73cd5a152b3f03b24cc"
Oct 30 13:36:12.037040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:12.037024    4291 scope.go:122] "RemoveContainer" containerID="2e6f178be130ce87068a0216d4aa0cb862c88162940ea28e66ca19becd1a1410"
Oct 30 13:36:16.276625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:16.276595    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 13:36:16.277097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:16.276720    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 13:36:16.739010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:16.738968    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"]
Oct 30 13:36:16.739257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:16.739230    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy" containerID="cri-o://0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8" gracePeriod=30
Oct 30 13:36:17.024711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.024680    4291 patch_prober.go:29] interesting pod/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused" start-of-body=
Oct 30 13:36:17.024855 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.024750    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.87:15021/healthz/ready\": dial tcp 10.133.0.87:15021: connect: connection refused"
Oct 30 13:36:17.343877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.343057    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:36:17.356414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.356378    4291 generic.go:356] "Generic (PLEG): container finished" podID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerID="a20baaf687e71cf58e1e1710972e314913f7d7e0b79704215983041dfe20063c" exitCode=0
Oct 30 13:36:17.356595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.356540    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" event={"ID":"0cdff5ab-53ed-4c01-9858-d9f97611dbbf","Type":"ContainerDied","Data":"a20baaf687e71cf58e1e1710972e314913f7d7e0b79704215983041dfe20063c"}
Oct 30 13:36:17.359181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.358818    4291 generic.go:356] "Generic (PLEG): container finished" podID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerID="458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15" exitCode=0
Oct 30 13:36:17.359181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.358901    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" event={"ID":"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3","Type":"ContainerDied","Data":"458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15"}
Oct 30 13:36:17.359181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.358934    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq" event={"ID":"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3","Type":"ContainerDied","Data":"61a9a02f06f0f1b0fa01afc7dba3cb433d515635d2773378102f1c7b27b1341e"}
Oct 30 13:36:17.359181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.358955    4291 scope.go:122] "RemoveContainer" containerID="458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15"
Oct 30 13:36:17.359181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.359072    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"
Oct 30 13:36:17.361252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.361124    4291 generic.go:356] "Generic (PLEG): container finished" podID="9cb73da3-d431-4205-9dce-234b12384bb5" containerID="514a23ba4d8550e36f82910d7d3a5a86fe6fad422db32bfb4d19a69889ea574b" exitCode=0
Oct 30 13:36:17.361252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.361205    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" event={"ID":"9cb73da3-d431-4205-9dce-234b12384bb5","Type":"ContainerDied","Data":"514a23ba4d8550e36f82910d7d3a5a86fe6fad422db32bfb4d19a69889ea574b"}
Oct 30 13:36:17.374561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.374534    4291 scope.go:122] "RemoveContainer" containerID="458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15"
Oct 30 13:36:17.374994 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:36:17.374963    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15\": container with ID starting with 458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15 not found: ID does not exist" containerID="458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15"
Oct 30 13:36:17.375107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.375003    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15"} err="failed to get container status \"458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15\": rpc error: code = NotFound desc = could not find container \"458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15\": container with ID starting with 458c3d43238f982da65b9b4873bdd2d7c80ef117c5c3559ab6719578b7668f15 not found: ID does not exist"
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.420894    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-data\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.420995    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-socket\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.421044    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-certs\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.421090    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-token\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-token\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.421132    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-credential-socket\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.421190    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-kube-api-access-zhfg4\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-kube-api-access-zhfg4\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.421274    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-envoy\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.421430    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-podinfo\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.422661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.421504    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istiod-ca-cert\") pod \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\" (UID: \"9196df4b-d6bc-4ed9-8a50-dc8f828d48b3\") "
Oct 30 13:36:17.426103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.426069    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-credential-socket" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.426991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.426670    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-data" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.426991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.426872    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-socket" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.426991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.426920    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-certs" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.428393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.428274    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istiod-ca-cert" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:17.428896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.428824    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-kube-api-access-zhfg4" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "kube-api-access-zhfg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:17.430869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.430670    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-envoy" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.431740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.431717    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-podinfo" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:17.443018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.435596    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:36:17.443018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.436008    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-token" pod "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" (UID: "9196df4b-d6bc-4ed9-8a50-dc8f828d48b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:17.466062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.466029    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:36:17.522873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.522833    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-data\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.522933    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-kube-api-access-q2jmf\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-kube-api-access-q2jmf\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.523065    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cb73da3-d431-4205-9dce-234b12384bb5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb73da3-d431-4205-9dce-234b12384bb5-istiod-ca-cert\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.523157    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9cb73da3-d431-4205-9dce-234b12384bb5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb73da3-d431-4205-9dce-234b12384bb5-istio-podinfo\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.523191    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-credential-socket\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.523217    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-istio-token\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-istio-token\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.523259    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-certs\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.523443    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-envoy\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.523531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.523489    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-socket\") pod \"9cb73da3-d431-4205-9dce-234b12384bb5\" (UID: \"9cb73da3-d431-4205-9dce-234b12384bb5\") "
Oct 30 13:36:17.524085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524058    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhfg4\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-kube-api-access-zhfg4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524089    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524104    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524117    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524129    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524141    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524153    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524166    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524178    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.524667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524443    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-socket" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.524667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.524623    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-data" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.526080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.525560    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb73da3-d431-4205-9dce-234b12384bb5-istiod-ca-cert" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:17.526080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.525839    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-certs" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.529034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.529010    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-credential-socket" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.529559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.529525    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-kube-api-access-q2jmf" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "kube-api-access-q2jmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:17.532081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.532046    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-envoy" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.532302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.532274    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-istio-token" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:17.537744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.537673    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cb73da3-d431-4205-9dce-234b12384bb5-istio-podinfo" pod "9cb73da3-d431-4205-9dce-234b12384bb5" (UID: "9cb73da3-d431-4205-9dce-234b12384bb5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:17.625120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625080    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-socket\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625149    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-credential-socket\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625186    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istiod-ca-cert\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625207    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-podinfo\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625225    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-data\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625342    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-token\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-token\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625398    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-certs\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625420    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-socket" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.625582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625441    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-kube-api-access-rgbld\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-kube-api-access-rgbld\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625526    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-envoy\") pod \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\" (UID: \"0cdff5ab-53ed-4c01-9858-d9f97611dbbf\") "
Oct 30 13:36:17.625582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625560    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-credential-socket" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.625946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625680    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-certs" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.625946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.625704    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istiod-ca-cert" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:17.626191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626148    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-data" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.626191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626179    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626200    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626213    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626228    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2jmf\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-kube-api-access-q2jmf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626243    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626255    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626271    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626285    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb73da3-d431-4205-9dce-234b12384bb5-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626298    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb73da3-d431-4205-9dce-234b12384bb5-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626312    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626325    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb73da3-d431-4205-9dce-234b12384bb5-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626339    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb73da3-d431-4205-9dce-234b12384bb5-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.626355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.626351    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.627825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.627801    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-podinfo" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:17.628006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.627987    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-token" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:17.628268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.628240    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-envoy" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:17.628369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.628351    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-kube-api-access-rgbld" pod "0cdff5ab-53ed-4c01-9858-d9f97611dbbf" (UID: "0cdff5ab-53ed-4c01-9858-d9f97611dbbf"). InnerVolumeSpecName "kube-api-access-rgbld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:17.727425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.727387    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.727425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.727425    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.727425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.727438    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.727801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.727450    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.727801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.727463    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgbld\" (UniqueName: \"kubernetes.io/projected/0cdff5ab-53ed-4c01-9858-d9f97611dbbf-kube-api-access-rgbld\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:17.735715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.735677    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"]
Oct 30 13:36:17.739577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.739551    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xv5qq"]
Oct 30 13:36:17.884860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:17.884819    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" path="/var/lib/kubelet/pods/9196df4b-d6bc-4ed9-8a50-dc8f828d48b3/volumes"
Oct 30 13:36:18.368888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.368856    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"
Oct 30 13:36:18.368888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.368873    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t" event={"ID":"0cdff5ab-53ed-4c01-9858-d9f97611dbbf","Type":"ContainerDied","Data":"0e956b59190de15b355075bef3b358c7ad2e3502da24adc499cd4ed860b86240"}
Oct 30 13:36:18.369444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.368946    4291 scope.go:122] "RemoveContainer" containerID="a20baaf687e71cf58e1e1710972e314913f7d7e0b79704215983041dfe20063c"
Oct 30 13:36:18.371948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.371918    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms" event={"ID":"9cb73da3-d431-4205-9dce-234b12384bb5","Type":"ContainerDied","Data":"d649b94383782fd54ed154a7faf64de925f66d7e6cac24251f4df14167a11e88"}
Oct 30 13:36:18.372073 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.372003    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"
Oct 30 13:36:18.385309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.385291    4291 scope.go:122] "RemoveContainer" containerID="514a23ba4d8550e36f82910d7d3a5a86fe6fad422db32bfb4d19a69889ea574b"
Oct 30 13:36:18.419499 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.419477    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"]
Oct 30 13:36:18.431284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.431261    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-xlsms"]
Oct 30 13:36:18.455232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.455204    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"]
Oct 30 13:36:18.459054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:18.459030    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-52af49-openshift-default-67d7587989-9k27t"]
Oct 30 13:36:19.473365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.473332    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"]
Oct 30 13:36:19.473896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.473880    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy"
Oct 30 13:36:19.473957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.473897    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy"
Oct 30 13:36:19.473957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.473924    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy"
Oct 30 13:36:19.473957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.473929    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy"
Oct 30 13:36:19.474057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.473964    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy"
Oct 30 13:36:19.474057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.473969    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy"
Oct 30 13:36:19.474057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.474048    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" containerName="istio-proxy"
Oct 30 13:36:19.474143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.474057    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9196df4b-d6bc-4ed9-8a50-dc8f828d48b3" containerName="istio-proxy"
Oct 30 13:36:19.474143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.474067    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" containerName="istio-proxy"
Oct 30 13:36:19.478749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.478732    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.493511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.493482    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"]
Oct 30 13:36:19.501390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.501367    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:36:19.501390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.501368    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:36:19.501559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.501417    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:36:19.501559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.501466    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:36:19.648744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.648708    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h22gn\" (UniqueName: \"kubernetes.io/projected/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-kube-api-access-h22gn\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.648934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.648824    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.648934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.648908    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.750657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.750560    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h22gn\" (UniqueName: \"kubernetes.io/projected/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-kube-api-access-h22gn\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.750657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.750624    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.750885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.750703    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.751340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.751316    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.753195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.753164    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.769283 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.769261    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h22gn\" (UniqueName: \"kubernetes.io/projected/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-kube-api-access-h22gn\") pod \"maas-api-revoke-sub-68a86172-zjjms\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.793530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.793505    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:19.880861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.880823    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdff5ab-53ed-4c01-9858-d9f97611dbbf" path="/var/lib/kubelet/pods/0cdff5ab-53ed-4c01-9858-d9f97611dbbf/volumes"
Oct 30 13:36:19.881472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:19.881455    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cb73da3-d431-4205-9dce-234b12384bb5" path="/var/lib/kubelet/pods/9cb73da3-d431-4205-9dce-234b12384bb5/volumes"
Oct 30 13:36:20.098738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:20.098712    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"]
Oct 30 13:36:20.099690 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:20.099663    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d6b581a_3428_45e9_aa28_76b5e6aa4aac.slice/crio-70e8ee04230d7fdb3cfc0b3acbba0e18beb0d6f5f78814e163e335473d5621ab WatchSource:0}: Error finding container 70e8ee04230d7fdb3cfc0b3acbba0e18beb0d6f5f78814e163e335473d5621ab: Status 404 returned error can't find the container with id 70e8ee04230d7fdb3cfc0b3acbba0e18beb0d6f5f78814e163e335473d5621ab
Oct 30 13:36:20.386972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:20.386887    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms" event={"ID":"6d6b581a-3428-45e9-aa28-76b5e6aa4aac","Type":"ContainerStarted","Data":"1187ab5129d8be2dde694ca97a620e4e275415f7033a80acf9f9f1e9aa1535e8"}
Oct 30 13:36:20.386972 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:20.386951    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms" event={"ID":"6d6b581a-3428-45e9-aa28-76b5e6aa4aac","Type":"ContainerStarted","Data":"70e8ee04230d7fdb3cfc0b3acbba0e18beb0d6f5f78814e163e335473d5621ab"}
Oct 30 13:36:20.426432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:20.426383    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms" podStartSLOduration=1.426369587 podStartE2EDuration="1.426369587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:20.424388395 +0000 UTC m=+1990.684626629" watchObservedRunningTime="2026-10-30 13:36:20.426369587 +0000 UTC m=+1990.686607820"
Oct 30 13:36:21.394365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.394336    4291 generic.go:356] "Generic (PLEG): container finished" podID="6d6b581a-3428-45e9-aa28-76b5e6aa4aac" containerID="1187ab5129d8be2dde694ca97a620e4e275415f7033a80acf9f9f1e9aa1535e8" exitCode=0
Oct 30 13:36:21.394365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.394371    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms" event={"ID":"6d6b581a-3428-45e9-aa28-76b5e6aa4aac","Type":"ContainerDied","Data":"1187ab5129d8be2dde694ca97a620e4e275415f7033a80acf9f9f1e9aa1535e8"}
Oct 30 13:36:21.465666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.465619    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"]
Oct 30 13:36:21.469628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.469604    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.486913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.486502    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-844856\""
Oct 30 13:36:21.492328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.492292    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"]
Oct 30 13:36:21.499074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.499046    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"]
Oct 30 13:36:21.503097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.503074    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:21.523484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.523455    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"]
Oct 30 13:36:21.568138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.568111    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-844856-dockercfg-w4q5q\""
Oct 30 13:36:21.591251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.591213    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxjtb\" (UniqueName: \"kubernetes.io/projected/604135f2-4b64-4c25-980c-df8d4315a28e-kube-api-access-wxjtb\") pod \"payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.591589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.591509    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604135f2-4b64-4c25-980c-df8d4315a28e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.591842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.591797    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3f7ed-f161-49d2-8654-a4f45e615352-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:21.592059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.592021    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5wx\" (UniqueName: \"kubernetes.io/projected/cbb3f7ed-f161-49d2-8654-a4f45e615352-kube-api-access-px5wx\") pod \"payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:21.763675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.762203    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5wx\" (UniqueName: \"kubernetes.io/projected/cbb3f7ed-f161-49d2-8654-a4f45e615352-kube-api-access-px5wx\") pod \"payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:21.763675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.762467    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxjtb\" (UniqueName: \"kubernetes.io/projected/604135f2-4b64-4c25-980c-df8d4315a28e-kube-api-access-wxjtb\") pod \"payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.763675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.762515    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604135f2-4b64-4c25-980c-df8d4315a28e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.763675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.762560    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3f7ed-f161-49d2-8654-a4f45e615352-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:21.763675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.763518    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3f7ed-f161-49d2-8654-a4f45e615352-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:21.763675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.763564    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604135f2-4b64-4c25-980c-df8d4315a28e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.781485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.781452    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxjtb\" (UniqueName: \"kubernetes.io/projected/604135f2-4b64-4c25-980c-df8d4315a28e-kube-api-access-wxjtb\") pod \"payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.781601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.781563    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5wx\" (UniqueName: \"kubernetes.io/projected/cbb3f7ed-f161-49d2-8654-a4f45e615352-kube-api-access-px5wx\") pod \"payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:21.783481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.783459    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:21.817131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:21.817087    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:22.057430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.057402    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:36:22.105780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.105748    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"]
Oct 30 13:36:22.105780 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:22.105762    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod604135f2_4b64_4c25_980c_df8d4315a28e.slice/crio-ea8f8fb89f3df8249cff728b75b057ed3b1a75df33a9bcc80d46cf33b2a83872 WatchSource:0}: Error finding container ea8f8fb89f3df8249cff728b75b057ed3b1a75df33a9bcc80d46cf33b2a83872: Status 404 returned error can't find the container with id ea8f8fb89f3df8249cff728b75b057ed3b1a75df33a9bcc80d46cf33b2a83872
Oct 30 13:36:22.150258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.150227    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"]
Oct 30 13:36:22.152937 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:22.152908    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbb3f7ed_f161_49d2_8654_a4f45e615352.slice/crio-f68c8d79bc4beee1ccd5505fd0102b9a069e3e9327ea7d3c8df51d290e01b6ca WatchSource:0}: Error finding container f68c8d79bc4beee1ccd5505fd0102b9a069e3e9327ea7d3c8df51d290e01b6ca: Status 404 returned error can't find the container with id f68c8d79bc4beee1ccd5505fd0102b9a069e3e9327ea7d3c8df51d290e01b6ca
Oct 30 13:36:22.167442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.167413    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-kube-api-access-h98fn\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-kube-api-access-h98fn\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.167697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.167672    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-token\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-token\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.167803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.167714    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-podinfo\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.167803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.167781    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-data\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.167921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.167842    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istiod-ca-cert\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.167973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.167936    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-credential-socket\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.168031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.167984    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-socket\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.168091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168066    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-certs\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.168158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168132    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-data" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:22.168207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168163    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-envoy\") pod \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\" (UID: \"20686636-cecc-4cba-80f7-0d1d4bd8e1b7\") "
Oct 30 13:36:22.168434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168323    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-socket" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:22.168434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168340    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-certs" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:22.168434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168365    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-credential-socket" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:22.168434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168406    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istiod-ca-cert" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:22.168774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168621    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.168774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168677    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.168774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168690    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.168774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168704    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.168774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.168718    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.170738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.170629    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-envoy" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:36:22.170938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.170916    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-kube-api-access-h98fn" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "kube-api-access-h98fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:22.171763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.171734    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-podinfo" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:36:22.171902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.171762    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-token" pod "20686636-cecc-4cba-80f7-0d1d4bd8e1b7" (UID: "20686636-cecc-4cba-80f7-0d1d4bd8e1b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:22.270214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.270177    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.270214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.270208    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h98fn\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-kube-api-access-h98fn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.270417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.270224    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.270417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.270239    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20686636-cecc-4cba-80f7-0d1d4bd8e1b7-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.401922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.401884    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" event={"ID":"604135f2-4b64-4c25-980c-df8d4315a28e","Type":"ContainerStarted","Data":"c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74"}
Oct 30 13:36:22.402373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.401943    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" event={"ID":"604135f2-4b64-4c25-980c-df8d4315a28e","Type":"ContainerStarted","Data":"ea8f8fb89f3df8249cff728b75b057ed3b1a75df33a9bcc80d46cf33b2a83872"}
Oct 30 13:36:22.402373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.401964    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:22.403385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.403355    4291 generic.go:356] "Generic (PLEG): container finished" podID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerID="0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8" exitCode=0
Oct 30 13:36:22.403505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.403399    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"
Oct 30 13:36:22.403505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.403448    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" event={"ID":"20686636-cecc-4cba-80f7-0d1d4bd8e1b7","Type":"ContainerDied","Data":"0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8"}
Oct 30 13:36:22.403505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.403496    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl" event={"ID":"20686636-cecc-4cba-80f7-0d1d4bd8e1b7","Type":"ContainerDied","Data":"7569f094f30dfaf31fc9ad6d877f19b3afabf36d5de2aa79d83110a72d38f22e"}
Oct 30 13:36:22.403643 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.403522    4291 scope.go:122] "RemoveContainer" containerID="0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8"
Oct 30 13:36:22.405241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.405219    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" event={"ID":"cbb3f7ed-f161-49d2-8654-a4f45e615352","Type":"ContainerStarted","Data":"b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2"}
Oct 30 13:36:22.405339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.405260    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" event={"ID":"cbb3f7ed-f161-49d2-8654-a4f45e615352","Type":"ContainerStarted","Data":"f68c8d79bc4beee1ccd5505fd0102b9a069e3e9327ea7d3c8df51d290e01b6ca"}
Oct 30 13:36:22.414256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.414236    4291 scope.go:122] "RemoveContainer" containerID="0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8"
Oct 30 13:36:22.414523 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:36:22.414501    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8\": container with ID starting with 0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8 not found: ID does not exist" containerID="0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8"
Oct 30 13:36:22.414607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.414530    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8"} err="failed to get container status \"0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8\": rpc error: code = NotFound desc = could not find container \"0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8\": container with ID starting with 0f7c375513e1cc70d095c6a24d08bdbfc9adb82baa8a0b38f812ebbcdeb586e8 not found: ID does not exist"
Oct 30 13:36:22.454817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.454764    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" podStartSLOduration=1.454749663 podStartE2EDuration="1.454749663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:22.450665907 +0000 UTC m=+1992.710904132" watchObservedRunningTime="2026-10-30 13:36:22.454749663 +0000 UTC m=+1992.714987897"
Oct 30 13:36:22.499394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.499359    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"]
Oct 30 13:36:22.502383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.502354    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-wbwbl"]
Oct 30 13:36:22.536243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.536191    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" podStartSLOduration=1.53617575 podStartE2EDuration="1.53617575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:22.534506179 +0000 UTC m=+1992.794744414" watchObservedRunningTime="2026-10-30 13:36:22.53617575 +0000 UTC m=+1992.796413983"
Oct 30 13:36:22.544714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.544688    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:22.675708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.675593    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-kube-api-access-h22gn\" (UniqueName: \"kubernetes.io/projected/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-kube-api-access-h22gn\") pod \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") "
Oct 30 13:36:22.675708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.675668    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-openshift-ca\") pod \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") "
Oct 30 13:36:22.675963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.675748    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-xks-ca\") pod \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\" (UID: \"6d6b581a-3428-45e9-aa28-76b5e6aa4aac\") "
Oct 30 13:36:22.676112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.676087    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-openshift-ca" pod "6d6b581a-3428-45e9-aa28-76b5e6aa4aac" (UID: "6d6b581a-3428-45e9-aa28-76b5e6aa4aac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:22.676376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.676353    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.677720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.677696    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-kube-api-access-h22gn" pod "6d6b581a-3428-45e9-aa28-76b5e6aa4aac" (UID: "6d6b581a-3428-45e9-aa28-76b5e6aa4aac"). InnerVolumeSpecName "kube-api-access-h22gn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:22.677791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.677754    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-xks-ca" pod "6d6b581a-3428-45e9-aa28-76b5e6aa4aac" (UID: "6d6b581a-3428-45e9-aa28-76b5e6aa4aac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:22.777621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.777589    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h22gn\" (UniqueName: \"kubernetes.io/projected/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-kube-api-access-h22gn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:22.777621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:22.777620    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6d6b581a-3428-45e9-aa28-76b5e6aa4aac-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:23.412926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:23.412894    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"
Oct 30 13:36:23.413349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:23.412922    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms" event={"ID":"6d6b581a-3428-45e9-aa28-76b5e6aa4aac","Type":"ContainerDied","Data":"70e8ee04230d7fdb3cfc0b3acbba0e18beb0d6f5f78814e163e335473d5621ab"}
Oct 30 13:36:23.413349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:23.412977    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70e8ee04230d7fdb3cfc0b3acbba0e18beb0d6f5f78814e163e335473d5621ab"
Oct 30 13:36:23.414347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:23.414319    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:23.879815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:23.879779    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" path="/var/lib/kubelet/pods/20686636-cecc-4cba-80f7-0d1d4bd8e1b7/volumes"
Oct 30 13:36:29.821415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.821382    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"]
Oct 30 13:36:29.821960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.821945    4291 cpu_manager.go:414] "containerMap: removing container" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy"
Oct 30 13:36:29.822004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.821961    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy"
Oct 30 13:36:29.822004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.821995    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6d6b581a-3428-45e9-aa28-76b5e6aa4aac" containerName="revoke-keys"
Oct 30 13:36:29.822004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.822001    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d6b581a-3428-45e9-aa28-76b5e6aa4aac" containerName="revoke-keys"
Oct 30 13:36:29.822105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.822078    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="20686636-cecc-4cba-80f7-0d1d4bd8e1b7" containerName="istio-proxy"
Oct 30 13:36:29.822105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.822091    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d6b581a-3428-45e9-aa28-76b5e6aa4aac" containerName="revoke-keys"
Oct 30 13:36:29.829592 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.829547    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.841406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.841375    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"]
Oct 30 13:36:29.849737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.849612    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:36:29.849925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.849789    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:36:29.849925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.849830    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:36:29.855813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.855785    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:36:29.877213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.877177    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nr6zz\" (UniqueName: \"kubernetes.io/projected/fd3f4b78-9725-49e5-9427-0a1f974e898a-kube-api-access-nr6zz\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.877430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.877297    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.877505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.877429    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.979262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.979224    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.979433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.979294    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nr6zz\" (UniqueName: \"kubernetes.io/projected/fd3f4b78-9725-49e5-9427-0a1f974e898a-kube-api-access-nr6zz\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.979433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.979402    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.980022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.979997    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.981886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.981865    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:29.998397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:29.998362    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nr6zz\" (UniqueName: \"kubernetes.io/projected/fd3f4b78-9725-49e5-9427-0a1f974e898a-kube-api-access-nr6zz\") pod \"maas-api-revoke-sub-923e5b73-k8q8h\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:30.149270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:30.149235    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:30.459681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:30.459648    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"]
Oct 30 13:36:30.461888 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:30.461855    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd3f4b78_9725_49e5_9427_0a1f974e898a.slice/crio-ae33715cd7553b7d43f5382c29e3f326e3744e80c3fae16bbe1361b592ee9e42 WatchSource:0}: Error finding container ae33715cd7553b7d43f5382c29e3f326e3744e80c3fae16bbe1361b592ee9e42: Status 404 returned error can't find the container with id ae33715cd7553b7d43f5382c29e3f326e3744e80c3fae16bbe1361b592ee9e42
Oct 30 13:36:30.480880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:30.480840    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h" event={"ID":"fd3f4b78-9725-49e5-9427-0a1f974e898a","Type":"ContainerStarted","Data":"ae33715cd7553b7d43f5382c29e3f326e3744e80c3fae16bbe1361b592ee9e42"}
Oct 30 13:36:31.488429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:31.488392    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd3f4b78-9725-49e5-9427-0a1f974e898a" containerID="cbc6caeeb985d9df7aa6c3b9eeefd467b2feb8783b1b52cf1cd5215291fcc42e" exitCode=0
Oct 30 13:36:31.488814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:31.488471    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h" event={"ID":"fd3f4b78-9725-49e5-9427-0a1f974e898a","Type":"ContainerDied","Data":"cbc6caeeb985d9df7aa6c3b9eeefd467b2feb8783b1b52cf1cd5215291fcc42e"}
Oct 30 13:36:32.651095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.651070    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:32.711059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.711022    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd3f4b78-9725-49e5-9427-0a1f974e898a-kube-api-access-nr6zz\" (UniqueName: \"kubernetes.io/projected/fd3f4b78-9725-49e5-9427-0a1f974e898a-kube-api-access-nr6zz\") pod \"fd3f4b78-9725-49e5-9427-0a1f974e898a\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") "
Oct 30 13:36:32.711226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.711133    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-xks-ca\") pod \"fd3f4b78-9725-49e5-9427-0a1f974e898a\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") "
Oct 30 13:36:32.711226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.711190    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-openshift-ca\") pod \"fd3f4b78-9725-49e5-9427-0a1f974e898a\" (UID: \"fd3f4b78-9725-49e5-9427-0a1f974e898a\") "
Oct 30 13:36:32.711670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.711623    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-openshift-ca" pod "fd3f4b78-9725-49e5-9427-0a1f974e898a" (UID: "fd3f4b78-9725-49e5-9427-0a1f974e898a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:36:32.713425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.713392    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-xks-ca" pod "fd3f4b78-9725-49e5-9427-0a1f974e898a" (UID: "fd3f4b78-9725-49e5-9427-0a1f974e898a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:36:32.713686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.713617    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd3f4b78-9725-49e5-9427-0a1f974e898a-kube-api-access-nr6zz" pod "fd3f4b78-9725-49e5-9427-0a1f974e898a" (UID: "fd3f4b78-9725-49e5-9427-0a1f974e898a"). InnerVolumeSpecName "kube-api-access-nr6zz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:36:32.812941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.812839    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:32.812941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.812869    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd3f4b78-9725-49e5-9427-0a1f974e898a-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:32.812941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:32.812880    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nr6zz\" (UniqueName: \"kubernetes.io/projected/fd3f4b78-9725-49e5-9427-0a1f974e898a-kube-api-access-nr6zz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:36:33.417952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:33.417918    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 13:36:33.503369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:33.503340    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"
Oct 30 13:36:33.503566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:33.503372    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h" event={"ID":"fd3f4b78-9725-49e5-9427-0a1f974e898a","Type":"ContainerDied","Data":"ae33715cd7553b7d43f5382c29e3f326e3744e80c3fae16bbe1361b592ee9e42"}
Oct 30 13:36:33.503566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:33.503429    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae33715cd7553b7d43f5382c29e3f326e3744e80c3fae16bbe1361b592ee9e42"
Oct 30 13:36:34.425556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:34.425527    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 13:36:38.210495 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:36:38.210455    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:36:51.330766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.330727    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"]
Oct 30 13:36:51.331573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.331548    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd3f4b78-9725-49e5-9427-0a1f974e898a" containerName="revoke-keys"
Oct 30 13:36:51.331698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.331574    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd3f4b78-9725-49e5-9427-0a1f974e898a" containerName="revoke-keys"
Oct 30 13:36:51.331776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.331705    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd3f4b78-9725-49e5-9427-0a1f974e898a" containerName="revoke-keys"
Oct 30 13:36:51.336473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.336449    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.355052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.355021    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"]
Oct 30 13:36:51.362170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.362139    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-d2cecb-openshift-default-dockercfg-756s4\""
Oct 30 13:36:51.449030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.448994    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-token\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449055    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-data\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449179    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istiod-ca-cert\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449288 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449264    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qt2l\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-kube-api-access-8qt2l\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449333 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449318    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-certs\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449382    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-credential-socket\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449452    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-socket\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449552    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-envoy\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.449608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.449591    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-podinfo\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.551461    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-token\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.551534    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-data\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.551600    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istiod-ca-cert\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.552447    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-data\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.552618    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qt2l\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-kube-api-access-8qt2l\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.552686    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-certs\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.552719    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-credential-socket\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.552794    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-socket\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.552851    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-envoy\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.552881    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-podinfo\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.553702    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-certs\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.554543    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istiod-ca-cert\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.555156    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-socket\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.555657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.555423    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-credential-socket\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.567664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.558856    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-envoy\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.578658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.572599    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-podinfo\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.578658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.578457    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-token\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.582699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.581315    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qt2l\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-kube-api-access-8qt2l\") pod \"e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") " pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:51.655660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:51.654888    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:52.019257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:52.019224    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"]
Oct 30 13:36:52.019523 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:52.019494    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a4a2eb_20d2_48bf_afc8_cb6900350068.slice/crio-c3b3d9b92081ba88720c4ea573107136316d22ff539c4754a53c633c8ebceb04 WatchSource:0}: Error finding container c3b3d9b92081ba88720c4ea573107136316d22ff539c4754a53c633c8ebceb04: Status 404 returned error can't find the container with id c3b3d9b92081ba88720c4ea573107136316d22ff539c4754a53c633c8ebceb04
Oct 30 13:36:52.022408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:52.022374    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:52.022507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:52.022442    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:52.022507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:52.022470    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:52.654146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:52.654112    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" event={"ID":"d1a4a2eb-20d2-48bf-afc8-cb6900350068","Type":"ContainerStarted","Data":"1d43bfc97521fa6de08dbeeaf2bff1b1543abaaa3d3dee84a8f626bc97bcfd7d"}
Oct 30 13:36:52.654559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:52.654164    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" event={"ID":"d1a4a2eb-20d2-48bf-afc8-cb6900350068","Type":"ContainerStarted","Data":"c3b3d9b92081ba88720c4ea573107136316d22ff539c4754a53c633c8ebceb04"}
Oct 30 13:36:52.656510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:52.656491    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:53.663990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:53.663960    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:53.722578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:53.722529    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podStartSLOduration=2.722514543 podStartE2EDuration="2.722514543s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:52.717526676 +0000 UTC m=+2022.977764910" watchObservedRunningTime="2026-10-30 13:36:53.722514543 +0000 UTC m=+2023.982752777"
Oct 30 13:36:54.358359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.358323    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"]
Oct 30 13:36:54.363264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.363240    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.381764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.381717    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"]
Oct 30 13:36:54.387758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.387730    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-e3eb98-openshift-default-dockercfg-k2l28\""
Oct 30 13:36:54.404077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404043    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-certs\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-socket\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404268    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-envoy\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404335    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-credential-socket\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404414    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-data\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404480    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba32c77-b382-4ea2-b087-785f9335d376-istio-podinfo\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404513    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm48t\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-kube-api-access-nm48t\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404627    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba32c77-b382-4ea2-b087-785f9335d376-istiod-ca-cert\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.404809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.404684    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-istio-token\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.520087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.520057    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-certs\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.521436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.520756    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-socket\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.521816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.521283    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-socket\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.522164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.522135    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-envoy\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.522663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.522505    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-certs\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.522872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.522835    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-credential-socket\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.523103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.523086    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-data\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.523755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.523738    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba32c77-b382-4ea2-b087-785f9335d376-istio-podinfo\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.524011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.523994    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm48t\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-kube-api-access-nm48t\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.525182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.525165    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-credential-socket\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.525412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.525119    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba32c77-b382-4ea2-b087-785f9335d376-istiod-ca-cert\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.525597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.525581    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-istio-token\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.526793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.523665    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-data\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.528685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.527220    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba32c77-b382-4ea2-b087-785f9335d376-istiod-ca-cert\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.545365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.540721    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-envoy\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.545365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.541210    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba32c77-b382-4ea2-b087-785f9335d376-istio-podinfo\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.561668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.561571    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm48t\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-kube-api-access-nm48t\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.570666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.562130    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-istio-token\") pod \"e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") " pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:54.670351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.670265    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:54.671444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.671426    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:36:54.682845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:54.682822    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:55.014014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.013987    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"]
Oct 30 13:36:55.015810 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:55.015783    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba32c77_b382_4ea2_b087_785f9335d376.slice/crio-823d405353d0bd00354e81285950ade0e110b1485f2c44269405cc52e429b5ff WatchSource:0}: Error finding container 823d405353d0bd00354e81285950ade0e110b1485f2c44269405cc52e429b5ff: Status 404 returned error can't find the container with id 823d405353d0bd00354e81285950ade0e110b1485f2c44269405cc52e429b5ff
Oct 30 13:36:55.018964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.018920    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:55.019052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.019014    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:55.019096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.019060    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:36:55.677795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.677762    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" event={"ID":"aba32c77-b382-4ea2-b087-785f9335d376","Type":"ContainerStarted","Data":"9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016"}
Oct 30 13:36:55.678223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.677821    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" event={"ID":"aba32c77-b382-4ea2-b087-785f9335d376","Type":"ContainerStarted","Data":"823d405353d0bd00354e81285950ade0e110b1485f2c44269405cc52e429b5ff"}
Oct 30 13:36:55.683349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.683321    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:55.727784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:55.727727    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podStartSLOduration=1.7277121119999999 podStartE2EDuration="1.727712112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:55.724288235 +0000 UTC m=+2025.984526469" watchObservedRunningTime="2026-10-30 13:36:55.727712112 +0000 UTC m=+2025.987950345"
Oct 30 13:36:56.690382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:56.690352    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:57.124028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.123992    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"]
Oct 30 13:36:57.127986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.127965    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.146526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.146496    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-d2cecb\""
Oct 30 13:36:57.150663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.146724    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-d2cecb-dockercfg-k85j9\""
Oct 30 13:36:57.150663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.148363    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"]
Oct 30 13:36:57.172079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.172040    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmjdn\" (UniqueName: \"kubernetes.io/projected/67fb9d16-7c3e-4a74-93b6-0cede5462c07-kube-api-access-lmjdn\") pod \"payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.172374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.172350    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fb9d16-7c3e-4a74-93b6-0cede5462c07-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.177171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.177150    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"]
Oct 30 13:36:57.181704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.181684    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.198932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.198906    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"]
Oct 30 13:36:57.355659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.355245    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda850-fa06-40c9-914e-0ad7ecde464c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.355659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.355333    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fb9d16-7c3e-4a74-93b6-0cede5462c07-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.355659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.355387    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmjdn\" (UniqueName: \"kubernetes.io/projected/67fb9d16-7c3e-4a74-93b6-0cede5462c07-kube-api-access-lmjdn\") pod \"payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.355659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.355481    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n8l2\" (UniqueName: \"kubernetes.io/projected/52fda850-fa06-40c9-914e-0ad7ecde464c-kube-api-access-9n8l2\") pod \"payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.359626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.359596    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fb9d16-7c3e-4a74-93b6-0cede5462c07-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.389225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.389141    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmjdn\" (UniqueName: \"kubernetes.io/projected/67fb9d16-7c3e-4a74-93b6-0cede5462c07-kube-api-access-lmjdn\") pod \"payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.443734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.443702    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:57.457165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.457133    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n8l2\" (UniqueName: \"kubernetes.io/projected/52fda850-fa06-40c9-914e-0ad7ecde464c-kube-api-access-9n8l2\") pod \"payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.457306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.457291    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda850-fa06-40c9-914e-0ad7ecde464c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.458355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.458328    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda850-fa06-40c9-914e-0ad7ecde464c-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.476054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.476025    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n8l2\" (UniqueName: \"kubernetes.io/projected/52fda850-fa06-40c9-914e-0ad7ecde464c-kube-api-access-9n8l2\") pod \"payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.503561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.503050    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:57.690933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.690855    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:57.691935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.691919    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:36:57.749956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.749923    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"]
Oct 30 13:36:57.764123 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:57.764093    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67fb9d16_7c3e_4a74_93b6_0cede5462c07.slice/crio-b8a379b79a5b5a0da0c082b76c9320d45ac9f1e828b80531114b273aab27dfde WatchSource:0}: Error finding container b8a379b79a5b5a0da0c082b76c9320d45ac9f1e828b80531114b273aab27dfde: Status 404 returned error can't find the container with id b8a379b79a5b5a0da0c082b76c9320d45ac9f1e828b80531114b273aab27dfde
Oct 30 13:36:57.826490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:57.826461    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"]
Oct 30 13:36:57.832908 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:36:57.832875    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fda850_fa06_40c9_914e_0ad7ecde464c.slice/crio-6e8a4804a72ed64a2aadc9d2c140cce75cb88cb5c09fe3ed3e8c25c4bc031f24 WatchSource:0}: Error finding container 6e8a4804a72ed64a2aadc9d2c140cce75cb88cb5c09fe3ed3e8c25c4bc031f24: Status 404 returned error can't find the container with id 6e8a4804a72ed64a2aadc9d2c140cce75cb88cb5c09fe3ed3e8c25c4bc031f24
Oct 30 13:36:58.698174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.698142    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" event={"ID":"67fb9d16-7c3e-4a74-93b6-0cede5462c07","Type":"ContainerStarted","Data":"263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc"}
Oct 30 13:36:58.698623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.698199    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" event={"ID":"67fb9d16-7c3e-4a74-93b6-0cede5462c07","Type":"ContainerStarted","Data":"b8a379b79a5b5a0da0c082b76c9320d45ac9f1e828b80531114b273aab27dfde"}
Oct 30 13:36:58.698623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.698229    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:36:58.699901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.699873    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" event={"ID":"52fda850-fa06-40c9-914e-0ad7ecde464c","Type":"ContainerStarted","Data":"bbb555928d5525c3d395b64217abcb702e2efbf238bac0131ee4f785f7c7f701"}
Oct 30 13:36:58.699983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.699927    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" event={"ID":"52fda850-fa06-40c9-914e-0ad7ecde464c","Type":"ContainerStarted","Data":"6e8a4804a72ed64a2aadc9d2c140cce75cb88cb5c09fe3ed3e8c25c4bc031f24"}
Oct 30 13:36:58.700072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.700055    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:36:58.739575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.739532    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" podStartSLOduration=1.739518687 podStartE2EDuration="1.739518687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:58.737462548 +0000 UTC m=+2028.997700776" watchObservedRunningTime="2026-10-30 13:36:58.739518687 +0000 UTC m=+2028.999756920"
Oct 30 13:36:58.780850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:36:58.780812    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" podStartSLOduration=1.780798141 podStartE2EDuration="1.780798141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:36:58.777059343 +0000 UTC m=+2029.037297577" watchObservedRunningTime="2026-10-30 13:36:58.780798141 +0000 UTC m=+2029.041036375"
Oct 30 13:37:01.043275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.043243    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"]
Oct 30 13:37:01.047305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.047282    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.069601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.069553    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-e3eb98\""
Oct 30 13:37:01.086978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.086939    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"]
Oct 30 13:37:01.225655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.213514    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c272e255-8e03-4edb-8dd2-50b0fb50c19d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.225655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.213697    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg7m4\" (UniqueName: \"kubernetes.io/projected/c272e255-8e03-4edb-8dd2-50b0fb50c19d-kube-api-access-qg7m4\") pod \"payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.316710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.315629    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg7m4\" (UniqueName: \"kubernetes.io/projected/c272e255-8e03-4edb-8dd2-50b0fb50c19d-kube-api-access-qg7m4\") pod \"payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.316710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.315884    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c272e255-8e03-4edb-8dd2-50b0fb50c19d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.317000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.316856    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c272e255-8e03-4edb-8dd2-50b0fb50c19d-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.337489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.337452    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg7m4\" (UniqueName: \"kubernetes.io/projected/c272e255-8e03-4edb-8dd2-50b0fb50c19d-kube-api-access-qg7m4\") pod \"payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.362095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.362067    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:01.670653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.670615    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"]
Oct 30 13:37:01.671400 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:37:01.671372    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc272e255_8e03_4edb_8dd2_50b0fb50c19d.slice/crio-759e29307d9713281e28ee0d028e0f9584bca5e1d555c42ceedf79e756799cba WatchSource:0}: Error finding container 759e29307d9713281e28ee0d028e0f9584bca5e1d555c42ceedf79e756799cba: Status 404 returned error can't find the container with id 759e29307d9713281e28ee0d028e0f9584bca5e1d555c42ceedf79e756799cba
Oct 30 13:37:01.721707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:01.721669    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" event={"ID":"c272e255-8e03-4edb-8dd2-50b0fb50c19d","Type":"ContainerStarted","Data":"759e29307d9713281e28ee0d028e0f9584bca5e1d555c42ceedf79e756799cba"}
Oct 30 13:37:02.729838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:02.729794    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" event={"ID":"c272e255-8e03-4edb-8dd2-50b0fb50c19d","Type":"ContainerStarted","Data":"df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02"}
Oct 30 13:37:02.730284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:02.729887    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:02.767239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:02.767182    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" podStartSLOduration=1.767165502 podStartE2EDuration="1.767165502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:02.766849604 +0000 UTC m=+2033.027087838" watchObservedRunningTime="2026-10-30 13:37:02.767165502 +0000 UTC m=+2033.027403771"
Oct 30 13:37:08.210546 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:37:08.210508    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:37:09.709144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:09.709116    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 13:37:09.709668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:09.709186    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 13:37:11.937597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:11.937562    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:37:13.739244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:13.739212    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 13:37:21.248955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.248918    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"]
Oct 30 13:37:21.252707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.252690    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:21.270173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.270142    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"]
Oct 30 13:37:21.277775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.277742    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-74589c\""
Oct 30 13:37:21.277953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.277854    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-74589c-dockercfg-jcvd6\""
Oct 30 13:37:21.281846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.281811    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef8317a-7398-4a40-860e-a8f15026eaef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:21.282008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.281947    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5w86x\" (UniqueName: \"kubernetes.io/projected/1ef8317a-7398-4a40-860e-a8f15026eaef-kube-api-access-5w86x\") pod \"payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:21.410333 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.383448    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef8317a-7398-4a40-860e-a8f15026eaef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:21.414188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.414153    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5w86x\" (UniqueName: \"kubernetes.io/projected/1ef8317a-7398-4a40-860e-a8f15026eaef-kube-api-access-5w86x\") pod \"payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:21.415262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.412594    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef8317a-7398-4a40-860e-a8f15026eaef-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:21.443523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.443429    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5w86x\" (UniqueName: \"kubernetes.io/projected/1ef8317a-7398-4a40-860e-a8f15026eaef-kube-api-access-5w86x\") pod \"payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:21.566419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:21.566315    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:22.030657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:22.018076    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"]
Oct 30 13:37:22.917226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:22.917186    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" event={"ID":"1ef8317a-7398-4a40-860e-a8f15026eaef","Type":"ContainerStarted","Data":"ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9"}
Oct 30 13:37:22.917715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:22.917242    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" event={"ID":"1ef8317a-7398-4a40-860e-a8f15026eaef","Type":"ContainerStarted","Data":"90853881f602c0af98678460019c04aad9b95630d8597f2c84ba65ce5f81f5e1"}
Oct 30 13:37:22.917715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:22.917347    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:22.955486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:22.955440    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" podStartSLOduration=1.955428618 podStartE2EDuration="1.955428618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:22.95409862 +0000 UTC m=+2053.214336853" watchObservedRunningTime="2026-10-30 13:37:22.955428618 +0000 UTC m=+2053.215666851"
Oct 30 13:37:33.927522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:33.927493    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 13:37:38.210626 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:37:38.210587    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:37:39.808977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.808853    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"]
Oct 30 13:37:39.813823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.813792    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:39.840568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.840519    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-006c41e31efa\""
Oct 30 13:37:39.840926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.840904    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"kube-root-ca.crt\""
Oct 30 13:37:39.841047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.840982    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-plugins-e2e-extmod-b-844856\""
Oct 30 13:37:39.841117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.841052    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"routing-overlay\""
Oct 30 13:37:39.841200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.841184    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-e2e--cbdef895-dockercfg-rrsgw\""
Oct 30 13:37:39.844615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.844591    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"openshift-service-ca.crt\""
Oct 30 13:37:39.846516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.846492    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"]
Oct 30 13:37:39.901696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.901663    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:39.901862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.901720    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:39.901862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.901807    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:39.901943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:39.901924    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gb6lz\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-kube-api-access-gb6lz\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.003393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.003352    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.003603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.003458    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gb6lz\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-kube-api-access-gb6lz\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.003714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.003624    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.003714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.003688    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.004407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.004380    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.004536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.004481    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.006563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.006412    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.032657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.028495    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gb6lz\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-kube-api-access-gb6lz\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.133443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.132886    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:40.487976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:40.487942    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"]
Oct 30 13:37:40.489299 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:37:40.489242    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb66ba11d_2c71_406b_88e5_6947953bc88d.slice/crio-60b67d4a3bdfaef302631eb44f44aba8d355d53e751bae0a93ea2125c05bae28 WatchSource:0}: Error finding container 60b67d4a3bdfaef302631eb44f44aba8d355d53e751bae0a93ea2125c05bae28: Status 404 returned error can't find the container with id 60b67d4a3bdfaef302631eb44f44aba8d355d53e751bae0a93ea2125c05bae28
Oct 30 13:37:41.109667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:41.109613    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8" event={"ID":"b66ba11d-2c71-406b-88e5-6947953bc88d","Type":"ContainerStarted","Data":"50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8"}
Oct 30 13:37:41.110139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:41.109684    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8" event={"ID":"b66ba11d-2c71-406b-88e5-6947953bc88d","Type":"ContainerStarted","Data":"60b67d4a3bdfaef302631eb44f44aba8d355d53e751bae0a93ea2125c05bae28"}
Oct 30 13:37:41.110139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:41.109713    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:41.149823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:41.149773    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8" podStartSLOduration=2.149759754 podStartE2EDuration="2.149759754s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:37:41.146834122 +0000 UTC m=+2071.407072357" watchObservedRunningTime="2026-10-30 13:37:41.149759754 +0000 UTC m=+2071.409997988"
Oct 30 13:37:52.119413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:52.119381    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:37:56.243660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:37:56.243214    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:38:07.096588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.096544    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"]
Oct 30 13:38:07.123186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.123146    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.138771 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.130801    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"]
Oct 30 13:38:07.142808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.142708    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-c3efb2-openshift-default-dockercfg-s4rxl\""
Oct 30 13:38:07.167757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.167721    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istiod-ca-cert\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.167993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.167919    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skvzk\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-kube-api-access-skvzk\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.168090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.168054    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-certs\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.168149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.168099    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-podinfo\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.168207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.168157    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-credential-socket\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.168207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.168197    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-envoy\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.168311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.168224    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-socket\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.168397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.168381    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-token\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.168455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.168427    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-data\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.270795    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istiod-ca-cert\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.270982    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skvzk\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-kube-api-access-skvzk\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.271055    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-certs\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.271084    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-podinfo\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.271138    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-credential-socket\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.271184    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-envoy\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.271211    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-socket\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.271366    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-token\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.271404    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-data\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.272506    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-data\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.272852    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-socket\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.273304    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-certs\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.284657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.273926    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istiod-ca-cert\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.285746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.285554    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-credential-socket\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.302665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.289020    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-podinfo\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.318956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.318855    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-envoy\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.352737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.344475    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skvzk\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-kube-api-access-skvzk\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.352737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.345124    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-token\") pod \"e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") " pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.452355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.452295    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:07.812760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.812718    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"]
Oct 30 13:38:07.812897 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:38:07.812830    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd58119f_3ac9_4182_bbbc_48c1ae9fdb9a.slice/crio-5b79741186f445534753688da032f4d71fb29a73997c8ef132546b5131ee6d4f WatchSource:0}: Error finding container 5b79741186f445534753688da032f4d71fb29a73997c8ef132546b5131ee6d4f: Status 404 returned error can't find the container with id 5b79741186f445534753688da032f4d71fb29a73997c8ef132546b5131ee6d4f
Oct 30 13:38:07.815848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.815801    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:07.815982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.815883    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:07.815982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:07.815915    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:08.220570 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:38:08.220458    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:38:08.409470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.409435    4291 generic.go:356] "Generic (PLEG): container finished" podID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerID="1d43bfc97521fa6de08dbeeaf2bff1b1543abaaa3d3dee84a8f626bc97bcfd7d" exitCode=0
Oct 30 13:38:08.409709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.409508    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" event={"ID":"d1a4a2eb-20d2-48bf-afc8-cb6900350068","Type":"ContainerDied","Data":"1d43bfc97521fa6de08dbeeaf2bff1b1543abaaa3d3dee84a8f626bc97bcfd7d"}
Oct 30 13:38:08.410043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.410019    4291 scope.go:122] "RemoveContainer" containerID="1d43bfc97521fa6de08dbeeaf2bff1b1543abaaa3d3dee84a8f626bc97bcfd7d"
Oct 30 13:38:08.411333 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.411308    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" event={"ID":"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a","Type":"ContainerStarted","Data":"5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38"}
Oct 30 13:38:08.411435 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.411363    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" event={"ID":"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a","Type":"ContainerStarted","Data":"5b79741186f445534753688da032f4d71fb29a73997c8ef132546b5131ee6d4f"}
Oct 30 13:38:08.412207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.412173    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:08.412307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.412263    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:08.412362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.412312    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:08.413179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.413157    4291 generic.go:356] "Generic (PLEG): container finished" podID="aba32c77-b382-4ea2-b087-785f9335d376" containerID="9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016" exitCode=0
Oct 30 13:38:08.413282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.413191    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" event={"ID":"aba32c77-b382-4ea2-b087-785f9335d376","Type":"ContainerDied","Data":"9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016"}
Oct 30 13:38:08.413611 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.413591    4291 scope.go:122] "RemoveContainer" containerID="9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016"
Oct 30 13:38:08.415145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.415120    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:08.415658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.415185    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:08.415658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.415218    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:08.452791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.452769    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:08.461072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.461048    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:08.548737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:08.548670    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podStartSLOduration=1.548646063 podStartE2EDuration="1.548646063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:08.544625191 +0000 UTC m=+2098.804863425" watchObservedRunningTime="2026-10-30 13:38:08.548646063 +0000 UTC m=+2098.808884290"
Oct 30 13:38:09.421963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.421917    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" event={"ID":"aba32c77-b382-4ea2-b087-785f9335d376","Type":"ContainerStarted","Data":"7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636"}
Oct 30 13:38:09.424197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.424166    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" event={"ID":"d1a4a2eb-20d2-48bf-afc8-cb6900350068","Type":"ContainerStarted","Data":"d069b140075da6c91605cec6fb8fcae2df6fce6093314a37f4b6ee5fe9dddfef"}
Oct 30 13:38:09.424589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.424569    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:09.425583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.425563    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:38:09.675803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.673754    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:38:09.675803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.674190    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:38:09.695696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.694267    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:38:09.695696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.694686    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:38:09.695696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.694755    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:38:09.698877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.698737    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:38:09.698877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:09.698798    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:38:10.673661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:10.673072    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:38:10.673661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:10.673136    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:38:10.697660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:10.685051    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:38:10.697660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:10.685113    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:38:11.665660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:11.663095    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:38:11.665660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:11.663164    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:38:11.698661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:11.697793    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:38:11.698661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:11.697861    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:38:12.669661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:12.658179    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:38:12.669661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:12.658250    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:38:12.699661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:12.698879    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:38:12.699661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:12.699237    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:38:12.699661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:12.699291    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:38:13.664176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:13.663727    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:38:13.664176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:13.663795    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:38:13.704662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:13.696690    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:38:13.704662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:13.696783    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:38:14.079726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.078176    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"]
Oct 30 13:38:14.123545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.121964    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"]
Oct 30 13:38:14.123545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.122133    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.164704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.157311    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-c3efb2-dockercfg-ptvzj\""
Oct 30 13:38:14.180690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.173928    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-c3efb2\""
Oct 30 13:38:14.329129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.329077    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9k9j\" (UniqueName: \"kubernetes.io/projected/da77146e-4068-48f5-a507-addb8019e6f0-kube-api-access-l9k9j\") pod \"payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.329465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.329442    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da77146e-4068-48f5-a507-addb8019e6f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.431021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.430983    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9k9j\" (UniqueName: \"kubernetes.io/projected/da77146e-4068-48f5-a507-addb8019e6f0-kube-api-access-l9k9j\") pod \"payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.431021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.431022    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da77146e-4068-48f5-a507-addb8019e6f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.431890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.431869    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da77146e-4068-48f5-a507-addb8019e6f0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.485659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.485380    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9k9j\" (UniqueName: \"kubernetes.io/projected/da77146e-4068-48f5-a507-addb8019e6f0-kube-api-access-l9k9j\") pod \"payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.528776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.528732    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:14.702388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.702293    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused" start-of-body=
Oct 30 13:38:14.704658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.702738    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.196:15021/healthz/ready\": dial tcp 10.133.0.196:15021: connect: connection refused"
Oct 30 13:38:14.704658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.702665    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused" start-of-body=
Oct 30 13:38:14.704658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.702817    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.197:15021/healthz/ready\": dial tcp 10.133.0.197:15021: connect: connection refused"
Oct 30 13:38:14.936901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:14.936864    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"]
Oct 30 13:38:15.512193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.512144    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" event={"ID":"da77146e-4068-48f5-a507-addb8019e6f0","Type":"ContainerStarted","Data":"ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759"}
Oct 30 13:38:15.512367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.512233    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:15.512367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.512262    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" event={"ID":"da77146e-4068-48f5-a507-addb8019e6f0","Type":"ContainerStarted","Data":"99bfdefdc5a6fee28a3e4e1dce7a0717e122517e6fbe8b7c032907af19266d39"}
Oct 30 13:38:15.553147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.553091    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" podStartSLOduration=2.553076155 podStartE2EDuration="2.553076155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:15.550139999 +0000 UTC m=+2105.810378236" watchObservedRunningTime="2026-10-30 13:38:15.553076155 +0000 UTC m=+2105.813314388"
Oct 30 13:38:15.694586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.692921    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:38:15.696253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.696179    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 13:38:15.697926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.696654    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:38:15.698931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:15.698907    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 13:38:21.016073 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.016040    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"]
Oct 30 13:38:21.019217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.019202    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.040605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.040561    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"]
Oct 30 13:38:21.045708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.045683    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"routing-overlay\""
Oct 30 13:38:21.046712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.046690    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-48f199422eae\""
Oct 30 13:38:21.046980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.046966    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"kube-root-ca.crt\""
Oct 30 13:38:21.047157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.047144    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"openshift-service-ca.crt\""
Oct 30 13:38:21.047458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.047443    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-e2e--cb40fe81-dockercfg-5rb7k\""
Oct 30 13:38:21.047657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.047626    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d2cecb\""
Oct 30 13:38:21.178630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.176509    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.178630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.176598    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.202657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.201831    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.202657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.201933    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpg4s\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-kube-api-access-cpg4s\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.303154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.303056    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.303154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.303117    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.303409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.303202    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.303409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.303275    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpg4s\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-kube-api-access-cpg4s\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.303894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.303868    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.304021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.303939    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.305601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.305584    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.327029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.326996    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpg4s\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-kube-api-access-cpg4s\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.334862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.334838    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:21.654320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:21.652733    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"]
Oct 30 13:38:21.667996 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:38:21.667953    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8d66f05_9b2c_4cad_bd2d_a09ce86171aa.slice/crio-e51449bce9de4fbb0e7a83ed2f93dce690459c9cb771b0bb012736dd764f173b WatchSource:0}: Error finding container e51449bce9de4fbb0e7a83ed2f93dce690459c9cb771b0bb012736dd764f173b: Status 404 returned error can't find the container with id e51449bce9de4fbb0e7a83ed2f93dce690459c9cb771b0bb012736dd764f173b
Oct 30 13:38:22.623761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:22.623681    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl" event={"ID":"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa","Type":"ContainerStarted","Data":"a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019"}
Oct 30 13:38:22.623761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:22.623763    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:22.624249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:22.623787    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl" event={"ID":"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa","Type":"ContainerStarted","Data":"e51449bce9de4fbb0e7a83ed2f93dce690459c9cb771b0bb012736dd764f173b"}
Oct 30 13:38:22.664479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:22.664406    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl" podStartSLOduration=2.664385757 podStartE2EDuration="2.664385757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:22.662946495 +0000 UTC m=+2112.923184749" watchObservedRunningTime="2026-10-30 13:38:22.664385757 +0000 UTC m=+2112.924623994"
Oct 30 13:38:24.681294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:24.681259    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"]
Oct 30 13:38:24.681819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:24.681520    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy" containerID="cri-o://c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0" gracePeriod=30
Oct 30 13:38:26.522812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:26.522779    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 13:38:27.678167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:27.678134    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"]
Oct 30 13:38:27.678729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:27.678397    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" containerID="cri-o://4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb" gracePeriod=30
Oct 30 13:38:29.955923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:29.955857    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:38:29.957285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:29.957265    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:38:30.044530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044492    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-kube-api-access-smgsv\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-kube-api-access-smgsv\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.044754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044592    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca4d001-f934-405c-b02d-16e95de2cb55-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca4d001-f934-405c-b02d-16e95de2cb55-istiod-ca-cert\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.044754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044630    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-token\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-token\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.044885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044761    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-envoy\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.044885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044839    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-data\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.045009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044895    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-certs\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.045009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044944    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-credential-socket\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.045009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.044968    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca4d001-f934-405c-b02d-16e95de2cb55-istiod-ca-cert" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:30.045160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045111    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-socket\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.045357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045220    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-credential-socket" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:30.045357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045257    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-podinfo\") pod \"7ca4d001-f934-405c-b02d-16e95de2cb55\" (UID: \"7ca4d001-f934-405c-b02d-16e95de2cb55\") "
Oct 30 13:38:30.045564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045452    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-socket" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:30.045564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045466    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-data" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:30.045564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045478    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-certs" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:30.045993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045968    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.046092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.045997    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ca4d001-f934-405c-b02d-16e95de2cb55-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.046092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.046014    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.046092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.046028    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.046267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.046149    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.047342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.047312    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-envoy" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:30.047455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.047374    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-token" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:30.047455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.047403    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-kube-api-access-smgsv" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "kube-api-access-smgsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:30.047686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.047613    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-podinfo" pod "7ca4d001-f934-405c-b02d-16e95de2cb55" (UID: "7ca4d001-f934-405c-b02d-16e95de2cb55"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:38:30.147801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.147771    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smgsv\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-kube-api-access-smgsv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.147801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.147798    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.147801 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.147810    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.148045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.147818    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ca4d001-f934-405c-b02d-16e95de2cb55-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:30.683865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.683822    4291 generic.go:356] "Generic (PLEG): container finished" podID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerID="c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0" exitCode=0
Oct 30 13:38:30.684087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.683876    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"
Oct 30 13:38:30.684087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.683907    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" event={"ID":"7ca4d001-f934-405c-b02d-16e95de2cb55","Type":"ContainerDied","Data":"c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0"}
Oct 30 13:38:30.684087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.683973    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc" event={"ID":"7ca4d001-f934-405c-b02d-16e95de2cb55","Type":"ContainerDied","Data":"c194b06e3148452fa31de1e1034fe25b9509fcebd750b55abc72757e3aaba377"}
Oct 30 13:38:30.684087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.684003    4291 scope.go:122] "RemoveContainer" containerID="c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0"
Oct 30 13:38:30.695775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.695758    4291 scope.go:122] "RemoveContainer" containerID="c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0"
Oct 30 13:38:30.696090 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:38:30.696071    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0\": container with ID starting with c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0 not found: ID does not exist" containerID="c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0"
Oct 30 13:38:30.696157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.696098    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0"} err="failed to get container status \"c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0\": rpc error: code = NotFound desc = could not find container \"c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0\": container with ID starting with c7254c02e48474aca660de976d34104bcefaf14dda970cf29023660dc4b5b9f0 not found: ID does not exist"
Oct 30 13:38:30.747430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.747400    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"]
Oct 30 13:38:30.759503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:30.759476    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-2s9bc"]
Oct 30 13:38:31.880452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:31.880413    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" path="/var/lib/kubelet/pods/7ca4d001-f934-405c-b02d-16e95de2cb55/volumes"
Oct 30 13:38:32.728229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.728196    4291 patch_prober.go:29] interesting pod/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused" start-of-body=
Oct 30 13:38:32.728383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.728261    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.69:15021/healthz/ready\": dial tcp 10.133.0.69:15021: connect: connection refused"
Oct 30 13:38:32.955369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.955298    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:38:32.985018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.984984    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-envoy\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985031    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-credential-socket\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985109    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-podinfo\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985140    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-certs\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985168    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istiod-ca-cert\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985223    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-socket\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985253    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-data\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985291    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-kube-api-access-kjmbf\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-kube-api-access-kjmbf\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985397    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-credential-socket" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.985450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985421    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-token\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-token\") pod \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\" (UID: \"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44\") "
Oct 30 13:38:32.985766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985595    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-certs" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.985834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985801    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-data" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.985880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985831    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istiod-ca-cert" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:38:32.985966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.985945    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-socket" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.986687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.986368    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.986687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.986394    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.986687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.986409    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.986687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.986436    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.986687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.986465    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:32.987677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.987613    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-podinfo" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:38:32.987788 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.987745    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-envoy" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:38:32.987895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.987871    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-kube-api-access-kjmbf" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "kube-api-access-kjmbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:32.988340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:32.988316    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-token" pod "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" (UID: "cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:38:33.088311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.088260    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjmbf\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-kube-api-access-kjmbf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.088311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.088303    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.088311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.088317    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.088569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.088328    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:38:33.633497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.633470    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:38:33.710626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.710589    4291 generic.go:356] "Generic (PLEG): container finished" podID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerID="4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb" exitCode=0
Oct 30 13:38:33.710846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.710699    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"
Oct 30 13:38:33.710846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.710683    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" event={"ID":"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44","Type":"ContainerDied","Data":"4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb"}
Oct 30 13:38:33.710846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.710761    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l" event={"ID":"cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44","Type":"ContainerDied","Data":"c883c6de0994fd148ca2df6eda8bbaa8f3bd21f7580ecab4ecb448e6ef41731d"}
Oct 30 13:38:33.710846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.710801    4291 scope.go:122] "RemoveContainer" containerID="4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb"
Oct 30 13:38:33.722538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.722513    4291 scope.go:122] "RemoveContainer" containerID="4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb"
Oct 30 13:38:33.722906 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:38:33.722885    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb\": container with ID starting with 4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb not found: ID does not exist" containerID="4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb"
Oct 30 13:38:33.722971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.722917    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb"} err="failed to get container status \"4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb\": rpc error: code = NotFound desc = could not find container \"4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb\": container with ID starting with 4a1287c7aeddd2bc4d6c30fcf086b01aa46b2a401a763fc28886545c034a79fb not found: ID does not exist"
Oct 30 13:38:33.763607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.763576    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"]
Oct 30 13:38:33.773011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.772982    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-2pl8l"]
Oct 30 13:38:33.881030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:33.880994    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" path="/var/lib/kubelet/pods/cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44/volumes"
Oct 30 13:38:38.221279 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:38:38.221242    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:38:43.186040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.186004    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"]
Oct 30 13:38:43.186932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.186907    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy"
Oct 30 13:38:43.186932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.186931    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy"
Oct 30 13:38:43.187167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.186945    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy"
Oct 30 13:38:43.187167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.186954    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy"
Oct 30 13:38:43.187167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.187099    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb0a422-3cb8-4d3b-b56a-d86a0ad28e44" containerName="istio-proxy"
Oct 30 13:38:43.187167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.187117    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca4d001-f934-405c-b02d-16e95de2cb55" containerName="istio-proxy"
Oct 30 13:38:43.189810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.189790    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.212307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.212278    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"]
Oct 30 13:38:43.216994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.216967    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-98f1cc-openshift-default-dockercfg-6m8kr\""
Oct 30 13:38:43.334063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334022    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-credential-socket\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334085    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beeec12-22df-45bf-8423-6614d0f55596-istio-podinfo\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334248    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-istio-token\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334296    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-data\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334381    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-envoy\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334423    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beeec12-22df-45bf-8423-6614d0f55596-istiod-ca-cert\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334511    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk8fz\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-kube-api-access-tk8fz\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334568    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-certs\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.334663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.334648    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-socket\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439421    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-envoy\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439478    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beeec12-22df-45bf-8423-6614d0f55596-istiod-ca-cert\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439566    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk8fz\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-kube-api-access-tk8fz\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439617    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-certs\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439716    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-socket\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439791    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-credential-socket\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439863    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beeec12-22df-45bf-8423-6614d0f55596-istio-podinfo\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439922    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-istio-token\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.439950    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-data\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.440713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.440437    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-data\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.441425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.441041    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-socket\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.445597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.441727    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beeec12-22df-45bf-8423-6614d0f55596-istiod-ca-cert\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.445597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.442270    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-certs\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.445597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.443197    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-credential-socket\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.461716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.458479    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beeec12-22df-45bf-8423-6614d0f55596-istio-podinfo\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.469312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.469218    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk8fz\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-kube-api-access-tk8fz\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.473298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.472572    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-istio-token\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.478154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.478122    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-envoy\") pod \"e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") " pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.512201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.511907    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:43.896089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.896055    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"]
Oct 30 13:38:43.896977 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:38:43.896947    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7beeec12_22df_45bf_8423_6614d0f55596.slice/crio-3673f2ab6abef8f46402381a3ae80c1867ec379f31de9ee6ec3561953d952a5c WatchSource:0}: Error finding container 3673f2ab6abef8f46402381a3ae80c1867ec379f31de9ee6ec3561953d952a5c: Status 404 returned error can't find the container with id 3673f2ab6abef8f46402381a3ae80c1867ec379f31de9ee6ec3561953d952a5c
Oct 30 13:38:43.900340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.900300    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:43.900441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.900391    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:43.900479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:43.900436    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:38:44.804606 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:44.804568    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" event={"ID":"7beeec12-22df-45bf-8423-6614d0f55596","Type":"ContainerStarted","Data":"d09969c8c0d6d5d7c37cd7b1768fc6339e7a229b67135d4db9421b8ab9720ce8"}
Oct 30 13:38:44.805031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:44.804631    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" event={"ID":"7beeec12-22df-45bf-8423-6614d0f55596","Type":"ContainerStarted","Data":"3673f2ab6abef8f46402381a3ae80c1867ec379f31de9ee6ec3561953d952a5c"}
Oct 30 13:38:45.512620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:45.512580    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:45.519022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:45.518995    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:45.577125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:45.577075    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podStartSLOduration=2.577061372 podStartE2EDuration="2.577061372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:44.869704255 +0000 UTC m=+2135.129942489" watchObservedRunningTime="2026-10-30 13:38:45.577061372 +0000 UTC m=+2135.837299697"
Oct 30 13:38:45.810848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:45.810763    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:45.811916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:45.811896    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:38:49.095059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.095022    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"]
Oct 30 13:38:49.098649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.098606    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.116819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.116792    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"]
Oct 30 13:38:49.123916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.123876    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-98f1cc\""
Oct 30 13:38:49.124439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.124413    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-98f1cc-dockercfg-2kttz\""
Oct 30 13:38:49.263739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.261564    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqxp9\" (UniqueName: \"kubernetes.io/projected/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-kube-api-access-jqxp9\") pod \"payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.263739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.261854    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.314040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.306796    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"]
Oct 30 13:38:49.314040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.312756    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"]
Oct 30 13:38:49.314040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.312929    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.363009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.362928    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.363179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.363072    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqxp9\" (UniqueName: \"kubernetes.io/projected/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-kube-api-access-jqxp9\") pod \"payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.363816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.363788    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.382466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.382431    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqxp9\" (UniqueName: \"kubernetes.io/projected/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-kube-api-access-jqxp9\") pod \"payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.416643 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.416602    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.465002 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.464966    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n99np\" (UniqueName: \"kubernetes.io/projected/7fc89d43-2b87-4184-904e-8d86482b70a6-kube-api-access-n99np\") pod \"payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.465151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.465041    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc89d43-2b87-4184-904e-8d86482b70a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.566247 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.566203    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n99np\" (UniqueName: \"kubernetes.io/projected/7fc89d43-2b87-4184-904e-8d86482b70a6-kube-api-access-n99np\") pod \"payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.566453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.566276    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc89d43-2b87-4184-904e-8d86482b70a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.567023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.567000    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc89d43-2b87-4184-904e-8d86482b70a6-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.584840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.584803    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n99np\" (UniqueName: \"kubernetes.io/projected/7fc89d43-2b87-4184-904e-8d86482b70a6-kube-api-access-n99np\") pod \"payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.658883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.658787    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:49.725026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.724996    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"]
Oct 30 13:38:49.726278 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:38:49.726244    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcc28f6d_ac8b_4142_a4e3_ab628074f4dc.slice/crio-b82da46406ccff7b1aae7bd116b473ff1c12098b8d862ce57064c0e4cf8e241d WatchSource:0}: Error finding container b82da46406ccff7b1aae7bd116b473ff1c12098b8d862ce57064c0e4cf8e241d: Status 404 returned error can't find the container with id b82da46406ccff7b1aae7bd116b473ff1c12098b8d862ce57064c0e4cf8e241d
Oct 30 13:38:49.843591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.843557    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" event={"ID":"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc","Type":"ContainerStarted","Data":"1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d"}
Oct 30 13:38:49.843768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.843629    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:38:49.843768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.843658    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" event={"ID":"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc","Type":"ContainerStarted","Data":"b82da46406ccff7b1aae7bd116b473ff1c12098b8d862ce57064c0e4cf8e241d"}
Oct 30 13:38:49.882150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.882095    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" podStartSLOduration=0.88208043 podStartE2EDuration="882.08043ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:49.881448004 +0000 UTC m=+2140.141686238" watchObservedRunningTime="2026-10-30 13:38:49.88208043 +0000 UTC m=+2140.142318664"
Oct 30 13:38:49.974004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:49.973976    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"]
Oct 30 13:38:49.975149 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:38:49.975119    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc89d43_2b87_4184_904e_8d86482b70a6.slice/crio-5a0e2af0d68881856c1d4ab8d29a3002731b673f15e351d4f01015a2d90a1c85 WatchSource:0}: Error finding container 5a0e2af0d68881856c1d4ab8d29a3002731b673f15e351d4f01015a2d90a1c85: Status 404 returned error can't find the container with id 5a0e2af0d68881856c1d4ab8d29a3002731b673f15e351d4f01015a2d90a1c85
Oct 30 13:38:50.889751 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:50.889704    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" event={"ID":"7fc89d43-2b87-4184-904e-8d86482b70a6","Type":"ContainerStarted","Data":"8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c"}
Oct 30 13:38:50.891659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:50.891613    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" event={"ID":"7fc89d43-2b87-4184-904e-8d86482b70a6","Type":"ContainerStarted","Data":"5a0e2af0d68881856c1d4ab8d29a3002731b673f15e351d4f01015a2d90a1c85"}
Oct 30 13:38:50.891923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:50.891905    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:38:50.934409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:38:50.934347    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" podStartSLOduration=1.934331491 podStartE2EDuration="1.934331491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:38:50.931389836 +0000 UTC m=+2141.191628071" watchObservedRunningTime="2026-10-30 13:38:50.934331491 +0000 UTC m=+2141.194569747"
Oct 30 13:39:00.892335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:00.892304    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 13:39:01.900173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:01.900144    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 13:39:06.238225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:06.238185    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"]
Oct 30 13:39:06.256671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:06.256626    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:39:08.212624 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:39:08.212580    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:39:11.239121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.239063    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8" podUID="b66ba11d-2c71-406b-88e5-6947953bc88d" containerName="payload-processing" containerID="cri-o://50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8" gracePeriod=25
Oct 30 13:39:11.594984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.594954    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:39:11.696045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.696005    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-provider-credentials\") pod \"b66ba11d-2c71-406b-88e5-6947953bc88d\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") "
Oct 30 13:39:11.696256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.696119    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-plugins-config-volume\") pod \"b66ba11d-2c71-406b-88e5-6947953bc88d\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") "
Oct 30 13:39:11.696256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.696210    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-kube-api-access-gb6lz\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-kube-api-access-gb6lz\") pod \"b66ba11d-2c71-406b-88e5-6947953bc88d\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") "
Oct 30 13:39:11.696386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.696296    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-routing-overlay-volume\") pod \"b66ba11d-2c71-406b-88e5-6947953bc88d\" (UID: \"b66ba11d-2c71-406b-88e5-6947953bc88d\") "
Oct 30 13:39:11.696894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.696863    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-routing-overlay-volume" pod "b66ba11d-2c71-406b-88e5-6947953bc88d" (UID: "b66ba11d-2c71-406b-88e5-6947953bc88d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:11.697092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.697017    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-plugins-config-volume" pod "b66ba11d-2c71-406b-88e5-6947953bc88d" (UID: "b66ba11d-2c71-406b-88e5-6947953bc88d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:39:11.698600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.698572    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-kube-api-access-gb6lz" pod "b66ba11d-2c71-406b-88e5-6947953bc88d" (UID: "b66ba11d-2c71-406b-88e5-6947953bc88d"). InnerVolumeSpecName "kube-api-access-gb6lz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:11.698858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.698799    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-provider-credentials" pod "b66ba11d-2c71-406b-88e5-6947953bc88d" (UID: "b66ba11d-2c71-406b-88e5-6947953bc88d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:39:11.798009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.797906    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:39:11.798009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.797942    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gb6lz\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-kube-api-access-gb6lz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:39:11.798009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.797956    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b66ba11d-2c71-406b-88e5-6947953bc88d-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:39:11.798009 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:11.797967    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b66ba11d-2c71-406b-88e5-6947953bc88d-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:39:12.048946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.048854    4291 generic.go:356] "Generic (PLEG): container finished" podID="b66ba11d-2c71-406b-88e5-6947953bc88d" containerID="50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8" exitCode=0
Oct 30 13:39:12.048946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.048903    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"
Oct 30 13:39:12.048946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.048930    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8" event={"ID":"b66ba11d-2c71-406b-88e5-6947953bc88d","Type":"ContainerDied","Data":"50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8"}
Oct 30 13:39:12.049185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.048987    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8" event={"ID":"b66ba11d-2c71-406b-88e5-6947953bc88d","Type":"ContainerDied","Data":"60b67d4a3bdfaef302631eb44f44aba8d355d53e751bae0a93ea2125c05bae28"}
Oct 30 13:39:12.049185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.049010    4291 scope.go:122] "RemoveContainer" containerID="50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8"
Oct 30 13:39:12.084776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.084746    4291 scope.go:122] "RemoveContainer" containerID="50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8"
Oct 30 13:39:12.085107 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:39:12.085090    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8\": container with ID starting with 50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8 not found: ID does not exist" containerID="50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8"
Oct 30 13:39:12.085167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.085117    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8"} err="failed to get container status \"50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8\": rpc error: code = NotFound desc = could not find container \"50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8\": container with ID starting with 50b3d82a726947525e7553091cbd0b419cb04ddd7c07501c2e9626e15025eef8 not found: ID does not exist"
Oct 30 13:39:12.111792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.111753    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"]
Oct 30 13:39:12.117006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.116972    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cfmnkc8"]
Oct 30 13:39:12.204486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:12.204455    4291 scope.go:122] "RemoveContainer" containerID="76b976b67194b88329cf0859f206217b08ea78e42521ab75272c474d8f1638c4"
Oct 30 13:39:13.879734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:13.879700    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b66ba11d-2c71-406b-88e5-6947953bc88d" path="/var/lib/kubelet/pods/b66ba11d-2c71-406b-88e5-6947953bc88d/volumes"
Oct 30 13:39:17.109985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:17.109720    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerID="5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38" exitCode=0
Oct 30 13:39:17.110439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:17.109990    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" event={"ID":"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a","Type":"ContainerDied","Data":"5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38"}
Oct 30 13:39:17.110491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:17.110479    4291 scope.go:122] "RemoveContainer" containerID="5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38"
Oct 30 13:39:17.112500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:17.112467    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:39:17.112610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:17.112544    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:39:17.112610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:17.112581    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:39:18.119171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:18.119132    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" event={"ID":"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a","Type":"ContainerStarted","Data":"8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a"}
Oct 30 13:39:18.464727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:18.463129    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:39:18.464727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:18.463264    4291 patch_prober.go:29] interesting pod/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 13:39:18.464727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:18.463324    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 13:39:19.466661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:19.454087    4291 patch_prober.go:29] interesting pod/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 13:39:19.466661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:19.454168    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 13:39:20.455157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:20.455115    4291 patch_prober.go:29] interesting pod/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 13:39:20.455465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:20.455433    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 13:39:21.472660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:21.461096    4291 patch_prober.go:29] interesting pod/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 13:39:21.472660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:21.461177    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 13:39:22.474836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:22.474791    4291 patch_prober.go:29] interesting pod/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 13:39:22.475284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:22.474856    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 13:39:23.452875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:23.452841    4291 patch_prober.go:29] interesting pod/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused" start-of-body=
Oct 30 13:39:23.453065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:23.452902    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.203:15021/healthz/ready\": dial tcp 10.133.0.203:15021: connect: connection refused"
Oct 30 13:39:24.035992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:24.035957    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"]
Oct 30 13:39:24.047730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:24.047693    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47161c45-fnrlg"]
Oct 30 13:39:24.424649 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:24.424612    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:39:24.459261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:24.459229    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:39:25.186607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:25.186573    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 13:39:25.880673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:25.880622    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f063c777-7088-4e83-98e4-31f969253e46" path="/var/lib/kubelet/pods/f063c777-7088-4e83-98e4-31f969253e46/volumes"
Oct 30 13:39:34.041165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:34.041127    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"]
Oct 30 13:39:34.053327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:34.053290    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-50cb5d3b-zbpwh"]
Oct 30 13:39:35.880484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:35.880450    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f64a4c7-144d-426e-adce-e158ffac8c40" path="/var/lib/kubelet/pods/6f64a4c7-144d-426e-adce-e158ffac8c40/volumes"
Oct 30 13:39:36.795499 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:36.795463    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:39:38.211813 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:39:38.211775    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:39:39.778339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.778296    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"]
Oct 30 13:39:39.779171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.779152    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b66ba11d-2c71-406b-88e5-6947953bc88d" containerName="payload-processing"
Oct 30 13:39:39.779224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.779172    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b66ba11d-2c71-406b-88e5-6947953bc88d" containerName="payload-processing"
Oct 30 13:39:39.779304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.779292    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b66ba11d-2c71-406b-88e5-6947953bc88d" containerName="payload-processing"
Oct 30 13:39:39.782305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.782281    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.802144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.797120    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-a75d02-openshift-default-dockercfg-d6rrv\""
Oct 30 13:39:39.812851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.812816    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"]
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.921956    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-token\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922022    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-data\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922072    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-podinfo\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922159    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-certs\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922187    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjrs7\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-kube-api-access-vjrs7\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922314    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-credential-socket\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922365    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istiod-ca-cert\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922481    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-socket\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:39.932658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:39.922514    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-envoy\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.024517    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-certs\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.024571    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjrs7\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-kube-api-access-vjrs7\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.024723    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-credential-socket\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.024761    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istiod-ca-cert\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.024857    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-socket\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.024888    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-envoy\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.024986    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-token\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.025027    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-data\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.025661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.025072    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-podinfo\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.027217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.027186    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istiod-ca-cert\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.029736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.028103    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-socket\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.029736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.028364    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-credential-socket\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.029736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.029229    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-data\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.030188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.030164    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-certs\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.057414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.057380    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-token\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.073614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.073577    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjrs7\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-kube-api-access-vjrs7\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.083921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.083888    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-podinfo\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.084411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.084387    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-envoy\") pod \"e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") " pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.099973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.099934    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:40.451323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.451296    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"]
Oct 30 13:39:40.457479 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:39:40.456837    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded20293c_b4b6_4c37_b3b8_eecbaf8eef2b.slice/crio-39076feeeae2482039d9183e6028faeeb6f25c27d909115f6a7149452e3cf885 WatchSource:0}: Error finding container 39076feeeae2482039d9183e6028faeeb6f25c27d909115f6a7149452e3cf885: Status 404 returned error can't find the container with id 39076feeeae2482039d9183e6028faeeb6f25c27d909115f6a7149452e3cf885
Oct 30 13:39:40.462545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.462499    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:39:40.462721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.462599    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:39:40.462721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:40.462659    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:39:41.303398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:41.303359    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" event={"ID":"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b","Type":"ContainerStarted","Data":"a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6"}
Oct 30 13:39:41.303896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:41.303431    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" event={"ID":"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b","Type":"ContainerStarted","Data":"39076feeeae2482039d9183e6028faeeb6f25c27d909115f6a7149452e3cf885"}
Oct 30 13:39:42.101739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.101698    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:42.109105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.109075    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:42.175279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.175223    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podStartSLOduration=3.175204328 podStartE2EDuration="3.175204328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:41.359909972 +0000 UTC m=+2191.620148206" watchObservedRunningTime="2026-10-30 13:39:42.175204328 +0000 UTC m=+2192.435442562"
Oct 30 13:39:42.310320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.310287    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:42.311456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.311429    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:39:42.473500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.473414    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"]
Oct 30 13:39:42.477265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.477240    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.503156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.503122    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-plugins-e2e-extmod-b-e3eb98\""
Oct 30 13:39:42.506992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.506860    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-98ec1e033fce\""
Oct 30 13:39:42.506992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.506903    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"openshift-service-ca.crt\""
Oct 30 13:39:42.506992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.506943    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"agc-cred-94a5564a5433\""
Oct 30 13:39:42.508477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.507382    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"routing-overlay\""
Oct 30 13:39:42.509863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.509250    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"kube-root-ca.crt\""
Oct 30 13:39:42.510282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.510254    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-e3eb98\"/\"payload-processing-external-model-e2e--2547b850-dockercfg-rj6kb\""
Oct 30 13:39:42.514228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.514206    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"]
Oct 30 13:39:42.609585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.609549    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.609797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.609628    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.609797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.609734    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.610030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.609869    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4l7cr\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-kube-api-access-4l7cr\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.712659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.712040    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.712659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.712180    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.712659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.712226    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4l7cr\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-kube-api-access-4l7cr\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.712659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.712343    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.735934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.714327    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.735934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.716024    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.756875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.754813    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4l7cr\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-kube-api-access-4l7cr\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.761043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.757684    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") " pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:42.814483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:42.814174    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:43.131598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:43.131569    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"]
Oct 30 13:39:43.134442 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:39:43.134406    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4edcd9f1_9bea_456a_b298_828f5b727341.slice/crio-44c5496a848fb0fd76bc2685d48c9cec96f45fb0d19b6e7848703f3b3762b03e WatchSource:0}: Error finding container 44c5496a848fb0fd76bc2685d48c9cec96f45fb0d19b6e7848703f3b3762b03e: Status 404 returned error can't find the container with id 44c5496a848fb0fd76bc2685d48c9cec96f45fb0d19b6e7848703f3b3762b03e
Oct 30 13:39:43.336106 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:43.336069    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw" event={"ID":"4edcd9f1-9bea-456a-b298-828f5b727341","Type":"ContainerStarted","Data":"2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d"}
Oct 30 13:39:43.337016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:43.336747    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw" event={"ID":"4edcd9f1-9bea-456a-b298-828f5b727341","Type":"ContainerStarted","Data":"44c5496a848fb0fd76bc2685d48c9cec96f45fb0d19b6e7848703f3b3762b03e"}
Oct 30 13:39:43.337319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:43.337273    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:44.204766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.204699    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw" podStartSLOduration=2.204678215 podStartE2EDuration="2.204678215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:43.385941242 +0000 UTC m=+2193.646179475" watchObservedRunningTime="2026-10-30 13:39:44.204678215 +0000 UTC m=+2194.464916452"
Oct 30 13:39:44.207274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.207245    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"]
Oct 30 13:39:44.211827 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.211802    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.229146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.229115    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-0f92d2a185e9\""
Oct 30 13:39:44.233214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.231033    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"]
Oct 30 13:39:44.253332 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.253293    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c88qk\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-kube-api-access-c88qk\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.253555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.253532    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.253685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.253664    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.253761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.253746    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.355103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.355065    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.355607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.355185    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.355607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.355309    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c88qk\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-kube-api-access-c88qk\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.355607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.355568    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.355952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.355923    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.356089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.356027    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.358094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.358073    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.374964 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.374931    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c88qk\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-kube-api-access-c88qk\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.537818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.537713    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:44.864033 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:44.864004    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"]
Oct 30 13:39:44.865214 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:39:44.865188    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb90a767a_9e09_4a31_8635_510f625163dd.slice/crio-58418a407901cc167884242d8aee0511a8cf148b2e82635c64337e7728bd9c3f WatchSource:0}: Error finding container 58418a407901cc167884242d8aee0511a8cf148b2e82635c64337e7728bd9c3f: Status 404 returned error can't find the container with id 58418a407901cc167884242d8aee0511a8cf148b2e82635c64337e7728bd9c3f
Oct 30 13:39:45.349283 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:45.349250    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc" event={"ID":"b90a767a-9e09-4a31-8635-510f625163dd","Type":"ContainerStarted","Data":"5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296"}
Oct 30 13:39:45.349573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:45.349304    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc" event={"ID":"b90a767a-9e09-4a31-8635-510f625163dd","Type":"ContainerStarted","Data":"58418a407901cc167884242d8aee0511a8cf148b2e82635c64337e7728bd9c3f"}
Oct 30 13:39:45.349573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:45.349546    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:45.395692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:45.390618    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc" podStartSLOduration=1.3905920649999999 podStartE2EDuration="1.390592065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:39:45.388706906 +0000 UTC m=+2195.648945141" watchObservedRunningTime="2026-10-30 13:39:45.390592065 +0000 UTC m=+2195.650830301"
Oct 30 13:39:55.353187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:55.353110    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:39:56.360479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:56.360449    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:39:56.435131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:39:56.435093    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"]
Oct 30 13:40:01.436966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.436897    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl" podUID="f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" containerName="payload-processing" containerID="cri-o://a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019" gracePeriod=25
Oct 30 13:40:01.851589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.850770    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:40:01.929755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.917837    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-provider-credentials\") pod \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") "
Oct 30 13:40:01.929755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.918080    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-kube-api-access-cpg4s\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-kube-api-access-cpg4s\") pod \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") "
Oct 30 13:40:01.929755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.918293    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-plugins-config-volume\") pod \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") "
Oct 30 13:40:01.929755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.918335    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-routing-overlay-volume\") pod \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\" (UID: \"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa\") "
Oct 30 13:40:01.929755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.922409    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-plugins-config-volume" pod "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" (UID: "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:01.939584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:01.939539    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-routing-overlay-volume" pod "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" (UID: "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:02.013422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.006360    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:02.013422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.006400    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:02.023255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.023197    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-kube-api-access-cpg4s" pod "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" (UID: "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa"). InnerVolumeSpecName "kube-api-access-cpg4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:02.023540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.023524    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-provider-credentials" pod "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" (UID: "f8d66f05-9b2c-4cad-bd2d-a09ce86171aa"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:02.107656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.107604    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:02.107656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.107654    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cpg4s\" (UniqueName: \"kubernetes.io/projected/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa-kube-api-access-cpg4s\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:02.496420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.496319    4291 generic.go:356] "Generic (PLEG): container finished" podID="f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" containerID="a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019" exitCode=0
Oct 30 13:40:02.496420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.496363    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"
Oct 30 13:40:02.496420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.496394    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl" event={"ID":"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa","Type":"ContainerDied","Data":"a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019"}
Oct 30 13:40:02.497071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.496431    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl" event={"ID":"f8d66f05-9b2c-4cad-bd2d-a09ce86171aa","Type":"ContainerDied","Data":"e51449bce9de4fbb0e7a83ed2f93dce690459c9cb771b0bb012736dd764f173b"}
Oct 30 13:40:02.497071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.496455    4291 scope.go:122] "RemoveContainer" containerID="a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019"
Oct 30 13:40:02.508905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.508877    4291 scope.go:122] "RemoveContainer" containerID="a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019"
Oct 30 13:40:02.509295 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:40:02.509267    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019\": container with ID starting with a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019 not found: ID does not exist" containerID="a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019"
Oct 30 13:40:02.509359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.509311    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019"} err="failed to get container status \"a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019\": rpc error: code = NotFound desc = could not find container \"a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019\": container with ID starting with a2d268c9a1a05536211ce09c6d942105e02f606e675ddc5031a57a68020cc019 not found: ID does not exist"
Oct 30 13:40:02.548071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.548036    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"]
Oct 30 13:40:02.568661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:02.567751    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fmn4nl"]
Oct 30 13:40:03.884713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:03.884676    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" path="/var/lib/kubelet/pods/f8d66f05-9b2c-4cad-bd2d-a09ce86171aa/volumes"
Oct 30 13:40:08.216183 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:40:08.216138    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:40:10.556210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:10.556124    4291 generic.go:356] "Generic (PLEG): container finished" podID="7beeec12-22df-45bf-8423-6614d0f55596" containerID="d09969c8c0d6d5d7c37cd7b1768fc6339e7a229b67135d4db9421b8ab9720ce8" exitCode=0
Oct 30 13:40:10.556680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:10.556199    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" event={"ID":"7beeec12-22df-45bf-8423-6614d0f55596","Type":"ContainerDied","Data":"d09969c8c0d6d5d7c37cd7b1768fc6339e7a229b67135d4db9421b8ab9720ce8"}
Oct 30 13:40:10.556734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:10.556675    4291 scope.go:122] "RemoveContainer" containerID="d09969c8c0d6d5d7c37cd7b1768fc6339e7a229b67135d4db9421b8ab9720ce8"
Oct 30 13:40:10.558404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:10.558377    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:40:10.558487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:10.558444    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:40:10.558487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:10.558478    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:40:11.580741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:11.580684    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" event={"ID":"7beeec12-22df-45bf-8423-6614d0f55596","Type":"ContainerStarted","Data":"f29254231edf6c54cd248b27171aaab8bfa765c24f65aa89ea8d910608ad4a2c"}
Oct 30 13:40:12.513382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:12.513338    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:40:12.514997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:12.514969    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 13:40:12.515131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:12.515022    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 13:40:13.513394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:13.513353    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 13:40:13.513969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:13.513419    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 13:40:14.513094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:14.513057    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 13:40:14.513294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:14.513117    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 13:40:15.512889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:15.512849    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 13:40:15.513404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:15.512913    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 13:40:15.811385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:15.811297    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:40:16.048314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:16.048279    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"]
Oct 30 13:40:16.076157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:16.076014    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"]
Oct 30 13:40:16.080588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:16.080552    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5243cc8-5swhb"]
Oct 30 13:40:16.083859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:16.083834    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ac9725aa-bf78b"]
Oct 30 13:40:16.513521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:16.513484    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused" start-of-body=
Oct 30 13:40:16.513987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:16.513561    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.206:15021/healthz/ready\": dial tcp 10.133.0.206:15021: connect: connection refused"
Oct 30 13:40:17.518130 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:17.518101    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:40:17.519347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:17.519323    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 13:40:17.879728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:17.879686    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a6095a1-318c-4365-8625-1a639cdd268b" path="/var/lib/kubelet/pods/1a6095a1-318c-4365-8625-1a639cdd268b/volumes"
Oct 30 13:40:17.880306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:17.880283    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc04780-ecf5-4600-9e56-5d6b0f5c42d6" path="/var/lib/kubelet/pods/3bc04780-ecf5-4600-9e56-5d6b0f5c42d6/volumes"
Oct 30 13:40:26.032508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:26.032461    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"]
Oct 30 13:40:26.043885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:26.043851    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ca2b64f-wnmcr"]
Oct 30 13:40:27.033525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:27.033479    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"]
Oct 30 13:40:27.056320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:27.056279    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95e51fcb-cbdzh"]
Oct 30 13:40:27.880553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:27.880517    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ce11af-deba-4546-99cd-c319034911c9" path="/var/lib/kubelet/pods/92ce11af-deba-4546-99cd-c319034911c9/volumes"
Oct 30 13:40:27.881036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:27.881020    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c024db84-2b2c-4e26-9638-56e764b23a48" path="/var/lib/kubelet/pods/c024db84-2b2c-4e26-9638-56e764b23a48/volumes"
Oct 30 13:40:30.218824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:30.218780    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:40:34.761173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.761137    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"]
Oct 30 13:40:34.762028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.762005    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" containerName="payload-processing"
Oct 30 13:40:34.762159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.762029    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" containerName="payload-processing"
Oct 30 13:40:34.762237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.762196    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8d66f05-9b2c-4cad-bd2d-a09ce86171aa" containerName="payload-processing"
Oct 30 13:40:34.767852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.767829    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:34.782759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.782726    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"]
Oct 30 13:40:34.805880 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.805842    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-0cb091f4ffff\""
Oct 30 13:40:34.939808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.937558    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:34.939808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.937803    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:34.939808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.937862    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:34.939808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:34.937984    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9j5l\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-kube-api-access-s9j5l\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.044858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.044755    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.045797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.045772    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.046124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.046105    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9j5l\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-kube-api-access-s9j5l\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.046309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.046280    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.047558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.047532    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.048187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.048135    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.077222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.077167    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.083667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.081589    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9j5l\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-kube-api-access-s9j5l\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.093660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.088997    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.471850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.471814    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"]
Oct 30 13:40:35.475516 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:40:35.475476    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6848e19_f9e3_4282_a19a_2262d9b72d17.slice/crio-155adf4862267a24980e2d3f4d5b7577c5ae85f97844aadea39cf5f3a001c398 WatchSource:0}: Error finding container 155adf4862267a24980e2d3f4d5b7577c5ae85f97844aadea39cf5f3a001c398: Status 404 returned error can't find the container with id 155adf4862267a24980e2d3f4d5b7577c5ae85f97844aadea39cf5f3a001c398
Oct 30 13:40:35.782615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.782511    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz" event={"ID":"b6848e19-f9e3-4282-a19a-2262d9b72d17","Type":"ContainerStarted","Data":"e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5"}
Oct 30 13:40:35.782615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.782570    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz" event={"ID":"b6848e19-f9e3-4282-a19a-2262d9b72d17","Type":"ContainerStarted","Data":"155adf4862267a24980e2d3f4d5b7577c5ae85f97844aadea39cf5f3a001c398"}
Oct 30 13:40:35.782615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.782598    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:35.825802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.825743    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz" podStartSLOduration=1.825726329 podStartE2EDuration="1.825726329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:35.822087339 +0000 UTC m=+2246.082325573" watchObservedRunningTime="2026-10-30 13:40:35.825726329 +0000 UTC m=+2246.085964562"
Oct 30 13:40:35.859870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.859835    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"]
Oct 30 13:40:35.864375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.864351    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:35.890943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.881401    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"openshift-service-ca.crt\""
Oct 30 13:40:35.890943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.890694    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-e2e--175bf124-dockercfg-hhl62\""
Oct 30 13:40:35.955659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.897890    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"routing-overlay\""
Oct 30 13:40:35.955659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.897997    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"agc-cred-66e931314666\""
Oct 30 13:40:35.955659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.898064    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-plugins-e2e-extmod-a-310546\""
Oct 30 13:40:35.955659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.898173    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"kube-root-ca.crt\""
Oct 30 13:40:35.955659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:35.944979    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"]
Oct 30 13:40:36.082732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.082315    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.082732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.082410    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.082732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.082539    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ws4p\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-kube-api-access-2ws4p\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.082732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.082685    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.184045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.184005    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.184260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.184066    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.184260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.184155    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ws4p\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-kube-api-access-2ws4p\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.184260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.184250    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.184965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.184940    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.185063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.185018    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.186770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.186746    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.203457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.203425    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ws4p\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-kube-api-access-2ws4p\") pod \"payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.252470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.252432    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.567779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.567746    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"]
Oct 30 13:40:36.569175 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:40:36.569136    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87e53d15_3013_4324_b155_69131f2c3c87.slice/crio-b64f1fba691a2acacce14e6ccf8be8310d5ccb1523b7f65d90541b276c5ac3a1 WatchSource:0}: Error finding container b64f1fba691a2acacce14e6ccf8be8310d5ccb1523b7f65d90541b276c5ac3a1: Status 404 returned error can't find the container with id b64f1fba691a2acacce14e6ccf8be8310d5ccb1523b7f65d90541b276c5ac3a1
Oct 30 13:40:36.791605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.791564    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4" event={"ID":"87e53d15-3013-4324-b155-69131f2c3c87","Type":"ContainerStarted","Data":"4a704085ae84a7536f973541d2ec714ae85687a19218fe24b7273c7b61244dac"}
Oct 30 13:40:36.792090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.791619    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4" event={"ID":"87e53d15-3013-4324-b155-69131f2c3c87","Type":"ContainerStarted","Data":"b64f1fba691a2acacce14e6ccf8be8310d5ccb1523b7f65d90541b276c5ac3a1"}
Oct 30 13:40:36.792090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.791837    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:36.833133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:36.833025    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4" podStartSLOduration=1.833010147 podStartE2EDuration="1.833010147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:36.832344816 +0000 UTC m=+2247.092583051" watchObservedRunningTime="2026-10-30 13:40:36.833010147 +0000 UTC m=+2247.093248382"
Oct 30 13:40:38.211867 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:40:38.211828    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:40:40.469413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.469368    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"]
Oct 30 13:40:40.476688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.476659    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.477495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.477463    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"]
Oct 30 13:40:40.503378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.503348    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-006c41e31efa\""
Oct 30 13:40:40.503764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.503692    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"kube-root-ca.crt\""
Oct 30 13:40:40.503764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.503757    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"routing-overlay\""
Oct 30 13:40:40.503963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.503723    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-e2e--cbdef895-dockercfg-rrsgw\""
Oct 30 13:40:40.503963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.503776    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"payload-processing-external-model-plugins-e2e-extmod-b-844856\""
Oct 30 13:40:40.503963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.503723    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"agc-cred-62b7f7e20837\""
Oct 30 13:40:40.503963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.503854    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-844856\"/\"openshift-service-ca.crt\""
Oct 30 13:40:40.545176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.545143    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gk6w\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-kube-api-access-9gk6w\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.545363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.545212    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.545363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.545343    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.545439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.545372    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.646325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.646286    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gk6w\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-kube-api-access-9gk6w\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.646530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.646362    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.646530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.646476    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.646530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.646505    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.647231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.647205    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.647231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.647227    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.648995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.648975    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.704655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.700472    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gk6w\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-kube-api-access-9gk6w\") pod \"payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:40.797704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:40.797606    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:41.123723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:41.117283    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"]
Oct 30 13:40:41.830788 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:41.830749    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w" event={"ID":"c40fa1e4-849d-4660-81f6-832b74efc24e","Type":"ContainerStarted","Data":"10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592"}
Oct 30 13:40:41.831341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:41.830819    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w" event={"ID":"c40fa1e4-849d-4660-81f6-832b74efc24e","Type":"ContainerStarted","Data":"38d0374ba432bf8d272aafa9ec7e91a66a6a5360c10b78a738fecb0519bc8f0d"}
Oct 30 13:40:41.831341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:41.831023    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:41.871157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:41.871098    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w" podStartSLOduration=1.871078948 podStartE2EDuration="1.871078948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:41.868782558 +0000 UTC m=+2252.129020808" watchObservedRunningTime="2026-10-30 13:40:41.871078948 +0000 UTC m=+2252.131317183"
Oct 30 13:40:42.521355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:42.521315    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"]
Oct 30 13:40:42.524384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:42.524356    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:46.806105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:46.806072    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:40:46.878224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:46.878186    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"]
Oct 30 13:40:47.522319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:47.522270    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4" podUID="87e53d15-3013-4324-b155-69131f2c3c87" containerName="payload-processing" containerID="cri-o://4a704085ae84a7536f973541d2ec714ae85687a19218fe24b7273c7b61244dac" gracePeriod=25
Oct 30 13:40:48.241660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.236624    4291 generic.go:356] "Generic (PLEG): container finished" podID="87e53d15-3013-4324-b155-69131f2c3c87" containerID="4a704085ae84a7536f973541d2ec714ae85687a19218fe24b7273c7b61244dac" exitCode=0
Oct 30 13:40:48.241660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.236888    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4" event={"ID":"87e53d15-3013-4324-b155-69131f2c3c87","Type":"ContainerDied","Data":"4a704085ae84a7536f973541d2ec714ae85687a19218fe24b7273c7b61244dac"}
Oct 30 13:40:48.281596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.271744    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:48.479294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.479204    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-provider-credentials\") pod \"87e53d15-3013-4324-b155-69131f2c3c87\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") "
Oct 30 13:40:48.479489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.479323    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-kube-api-access-2ws4p\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-kube-api-access-2ws4p\") pod \"87e53d15-3013-4324-b155-69131f2c3c87\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") "
Oct 30 13:40:48.479489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.479357    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-plugins-config-volume\") pod \"87e53d15-3013-4324-b155-69131f2c3c87\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") "
Oct 30 13:40:48.479610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.479592    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-routing-overlay-volume\") pod \"87e53d15-3013-4324-b155-69131f2c3c87\" (UID: \"87e53d15-3013-4324-b155-69131f2c3c87\") "
Oct 30 13:40:48.480221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.479877    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-plugins-config-volume" pod "87e53d15-3013-4324-b155-69131f2c3c87" (UID: "87e53d15-3013-4324-b155-69131f2c3c87"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:48.480340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.480224    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:48.480340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.480271    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-routing-overlay-volume" pod "87e53d15-3013-4324-b155-69131f2c3c87" (UID: "87e53d15-3013-4324-b155-69131f2c3c87"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:48.482157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.482133    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-provider-credentials" pod "87e53d15-3013-4324-b155-69131f2c3c87" (UID: "87e53d15-3013-4324-b155-69131f2c3c87"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:48.482261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.482169    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-kube-api-access-2ws4p" pod "87e53d15-3013-4324-b155-69131f2c3c87" (UID: "87e53d15-3013-4324-b155-69131f2c3c87"). InnerVolumeSpecName "kube-api-access-2ws4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:48.506420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.506376    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"]
Oct 30 13:40:48.581631 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.581593    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:48.581846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.581628    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ws4p\" (UniqueName: \"kubernetes.io/projected/87e53d15-3013-4324-b155-69131f2c3c87-kube-api-access-2ws4p\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:48.581846 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.581672    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/87e53d15-3013-4324-b155-69131f2c3c87-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:48.767051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.766960    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"]
Oct 30 13:40:48.767739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.767720    4291 cpu_manager.go:414] "containerMap: removing container" podUID="87e53d15-3013-4324-b155-69131f2c3c87" containerName="payload-processing"
Oct 30 13:40:48.767796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.767740    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87e53d15-3013-4324-b155-69131f2c3c87" containerName="payload-processing"
Oct 30 13:40:48.767882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.767872    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="87e53d15-3013-4324-b155-69131f2c3c87" containerName="payload-processing"
Oct 30 13:40:48.772024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.771997    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.774578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.774546    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"]
Oct 30 13:40:48.786995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.786967    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"e2e-cfl-e402e5-provider-cred\""
Oct 30 13:40:48.787162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.787091    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-wk62z\""
Oct 30 13:40:48.787324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.787299    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\""
Oct 30 13:40:48.787402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.787349    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\""
Oct 30 13:40:48.787402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.787391    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\""
Oct 30 13:40:48.787514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.787392    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\""
Oct 30 13:40:48.885740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.885701    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vz8fk\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-kube-api-access-vz8fk\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.885933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.885747    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.885933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.885845    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.886015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.885989    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.987274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.987240    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.987812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.987776    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vz8fk\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-kube-api-access-vz8fk\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.987977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.987861    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.987977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.987939    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.988113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.988057    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.988817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.988795    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:48.990262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:48.990245    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:49.007413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.007383    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vz8fk\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-kube-api-access-vz8fk\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:49.090816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.090722    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:49.246056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.246022    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4" event={"ID":"87e53d15-3013-4324-b155-69131f2c3c87","Type":"ContainerDied","Data":"b64f1fba691a2acacce14e6ccf8be8310d5ccb1523b7f65d90541b276c5ac3a1"}
Oct 30 13:40:49.246505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.246084    4291 scope.go:122] "RemoveContainer" containerID="4a704085ae84a7536f973541d2ec714ae85687a19218fe24b7273c7b61244dac"
Oct 30 13:40:49.246505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.246034    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"
Oct 30 13:40:49.298697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.298653    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"]
Oct 30 13:40:49.302967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.302898    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-7958m7nl4"]
Oct 30 13:40:49.397926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.397902    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"]
Oct 30 13:40:49.398843 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:40:49.398818    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54221ff6_6615_4dfd_8590_17b47a9121c9.slice/crio-f2d098933de5e19c624c471c5561ee50a94230336699bda89a999b606e13102c WatchSource:0}: Error finding container f2d098933de5e19c624c471c5561ee50a94230336699bda89a999b606e13102c: Status 404 returned error can't find the container with id f2d098933de5e19c624c471c5561ee50a94230336699bda89a999b606e13102c
Oct 30 13:40:49.881272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:49.881236    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87e53d15-3013-4324-b155-69131f2c3c87" path="/var/lib/kubelet/pods/87e53d15-3013-4324-b155-69131f2c3c87/volumes"
Oct 30 13:40:50.256505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:50.256424    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2" event={"ID":"54221ff6-6615-4dfd-8590-17b47a9121c9","Type":"ContainerStarted","Data":"d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f"}
Oct 30 13:40:50.256505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:50.256473    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2" event={"ID":"54221ff6-6615-4dfd-8590-17b47a9121c9","Type":"ContainerStarted","Data":"f2d098933de5e19c624c471c5561ee50a94230336699bda89a999b606e13102c"}
Oct 30 13:40:50.256505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:50.256506    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:40:50.296254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:50.296199    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2" podStartSLOduration=2.296185066 podStartE2EDuration="2.296185066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:50.295264188 +0000 UTC m=+2260.555502422" watchObservedRunningTime="2026-10-30 13:40:50.296185066 +0000 UTC m=+2260.556423300"
Oct 30 13:40:51.878484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:51.878438    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc" podUID="b90a767a-9e09-4a31-8635-510f625163dd" containerName="payload-processing" containerID="cri-o://5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296" gracePeriod=25
Oct 30 13:40:52.161070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.161043    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:40:52.231366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.231329    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-plugins-config-volume\") pod \"b90a767a-9e09-4a31-8635-510f625163dd\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") "
Oct 30 13:40:52.231586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.231430    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-provider-credentials\") pod \"b90a767a-9e09-4a31-8635-510f625163dd\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") "
Oct 30 13:40:52.231586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.231533    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-kube-api-access-c88qk\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-kube-api-access-c88qk\") pod \"b90a767a-9e09-4a31-8635-510f625163dd\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") "
Oct 30 13:40:52.231755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.231622    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-routing-overlay-volume\") pod \"b90a767a-9e09-4a31-8635-510f625163dd\" (UID: \"b90a767a-9e09-4a31-8635-510f625163dd\") "
Oct 30 13:40:52.231996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.231959    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-plugins-config-volume" pod "b90a767a-9e09-4a31-8635-510f625163dd" (UID: "b90a767a-9e09-4a31-8635-510f625163dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:52.232141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.232001    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-routing-overlay-volume" pod "b90a767a-9e09-4a31-8635-510f625163dd" (UID: "b90a767a-9e09-4a31-8635-510f625163dd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:52.232286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.232261    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.232416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.232289    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b90a767a-9e09-4a31-8635-510f625163dd-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.233926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.233898    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-provider-credentials" pod "b90a767a-9e09-4a31-8635-510f625163dd" (UID: "b90a767a-9e09-4a31-8635-510f625163dd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:52.234447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.234424    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-kube-api-access-c88qk" pod "b90a767a-9e09-4a31-8635-510f625163dd" (UID: "b90a767a-9e09-4a31-8635-510f625163dd"). InnerVolumeSpecName "kube-api-access-c88qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:52.280800 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.280760    4291 generic.go:356] "Generic (PLEG): container finished" podID="b90a767a-9e09-4a31-8635-510f625163dd" containerID="5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296" exitCode=0
Oct 30 13:40:52.280992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.280825    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"
Oct 30 13:40:52.280992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.280838    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc" event={"ID":"b90a767a-9e09-4a31-8635-510f625163dd","Type":"ContainerDied","Data":"5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296"}
Oct 30 13:40:52.280992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.280918    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc" event={"ID":"b90a767a-9e09-4a31-8635-510f625163dd","Type":"ContainerDied","Data":"58418a407901cc167884242d8aee0511a8cf148b2e82635c64337e7728bd9c3f"}
Oct 30 13:40:52.280992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.280942    4291 scope.go:122] "RemoveContainer" containerID="5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296"
Oct 30 13:40:52.297005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.296983    4291 scope.go:122] "RemoveContainer" containerID="5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296"
Oct 30 13:40:52.297351 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:40:52.297327    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296\": container with ID starting with 5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296 not found: ID does not exist" containerID="5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296"
Oct 30 13:40:52.297440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.297357    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296"} err="failed to get container status \"5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296\": rpc error: code = NotFound desc = could not find container \"5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296\": container with ID starting with 5edbc79172a57c589d28eeda1a3844fe1a4b6caf323309f1cb042290d2eb3296 not found: ID does not exist"
Oct 30 13:40:52.333750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.333712    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.333750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.333741    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c88qk\" (UniqueName: \"kubernetes.io/projected/b90a767a-9e09-4a31-8635-510f625163dd-kube-api-access-c88qk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:52.335476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.335449    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"]
Oct 30 13:40:52.338849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.338824    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5797zsnjc"]
Oct 30 13:40:52.898189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:52.898161    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:40:53.519682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.510714    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw" podUID="4edcd9f1-9bea-456a-b298-828f5b727341" containerName="payload-processing" containerID="cri-o://2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d" gracePeriod=25
Oct 30 13:40:53.756715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.752163    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"]
Oct 30 13:40:53.756715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.753214    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b90a767a-9e09-4a31-8635-510f625163dd" containerName="payload-processing"
Oct 30 13:40:53.756715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.753236    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b90a767a-9e09-4a31-8635-510f625163dd" containerName="payload-processing"
Oct 30 13:40:53.756715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.753399    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b90a767a-9e09-4a31-8635-510f625163dd" containerName="payload-processing"
Oct 30 13:40:53.757751 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.757731    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.773892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.773826    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"]
Oct 30 13:40:53.838519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.838484    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:40:53.856320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856293    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-provider-credentials\") pod \"4edcd9f1-9bea-456a-b298-828f5b727341\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") "
Oct 30 13:40:53.856488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856470    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-routing-overlay-volume\") pod \"4edcd9f1-9bea-456a-b298-828f5b727341\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") "
Oct 30 13:40:53.856531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856503    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-kube-api-access-4l7cr\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-kube-api-access-4l7cr\") pod \"4edcd9f1-9bea-456a-b298-828f5b727341\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") "
Oct 30 13:40:53.856587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856533    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-plugins-config-volume\") pod \"4edcd9f1-9bea-456a-b298-828f5b727341\" (UID: \"4edcd9f1-9bea-456a-b298-828f5b727341\") "
Oct 30 13:40:53.856857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856832    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.856953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856884    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.856953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856887    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-routing-overlay-volume" pod "4edcd9f1-9bea-456a-b298-828f5b727341" (UID: "4edcd9f1-9bea-456a-b298-828f5b727341"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:53.857023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.856990    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-plugins-config-volume" pod "4edcd9f1-9bea-456a-b298-828f5b727341" (UID: "4edcd9f1-9bea-456a-b298-828f5b727341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:40:53.857090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.857068    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.857138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.857121    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dh6st\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-kube-api-access-dh6st\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.857342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.857326    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.857428 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.857348    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4edcd9f1-9bea-456a-b298-828f5b727341-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.858557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.858534    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-provider-credentials" pod "4edcd9f1-9bea-456a-b298-828f5b727341" (UID: "4edcd9f1-9bea-456a-b298-828f5b727341"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:53.858557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.858545    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-kube-api-access-4l7cr" pod "4edcd9f1-9bea-456a-b298-828f5b727341" (UID: "4edcd9f1-9bea-456a-b298-828f5b727341"). InnerVolumeSpecName "kube-api-access-4l7cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:40:53.886910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.886872    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b90a767a-9e09-4a31-8635-510f625163dd" path="/var/lib/kubelet/pods/b90a767a-9e09-4a31-8635-510f625163dd/volumes"
Oct 30 13:40:53.958616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.958571    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.959154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.958629    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.959154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.958760    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.959154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.958812    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dh6st\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-kube-api-access-dh6st\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.959154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.958910    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4l7cr\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-kube-api-access-4l7cr\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.959154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.958926    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4edcd9f1-9bea-456a-b298-828f5b727341-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:40:53.959480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.959451    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.959547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.959527    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.961187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.961164    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:53.977509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:53.977484    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dh6st\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-kube-api-access-dh6st\") pod \"payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") " pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:54.080118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.080020    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:54.299748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.299715    4291 generic.go:356] "Generic (PLEG): container finished" podID="4edcd9f1-9bea-456a-b298-828f5b727341" containerID="2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d" exitCode=0
Oct 30 13:40:54.299933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.299759    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"
Oct 30 13:40:54.299933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.299788    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw" event={"ID":"4edcd9f1-9bea-456a-b298-828f5b727341","Type":"ContainerDied","Data":"2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d"}
Oct 30 13:40:54.299933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.299899    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw" event={"ID":"4edcd9f1-9bea-456a-b298-828f5b727341","Type":"ContainerDied","Data":"44c5496a848fb0fd76bc2685d48c9cec96f45fb0d19b6e7848703f3b3762b03e"}
Oct 30 13:40:54.299933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.299928    4291 scope.go:122] "RemoveContainer" containerID="2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d"
Oct 30 13:40:54.311005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.310982    4291 scope.go:122] "RemoveContainer" containerID="2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d"
Oct 30 13:40:54.311349 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:40:54.311329    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d\": container with ID starting with 2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d not found: ID does not exist" containerID="2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d"
Oct 30 13:40:54.311423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.311362    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d"} err="failed to get container status \"2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d\": rpc error: code = NotFound desc = could not find container \"2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d\": container with ID starting with 2e255e93f23964ab1f4a7ad16b9020df445fcd41131f2629fffc998e6698e81d not found: ID does not exist"
Oct 30 13:40:54.351426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.351391    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"]
Oct 30 13:40:54.355146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.355123    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-e3eb98/payload-processing-external-model-e2e-extmod-b-e3eb98-6d5bvzjhw"]
Oct 30 13:40:54.390773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:54.390744    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"]
Oct 30 13:40:54.392213 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:40:54.392188    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8413eea7_dfb7_44ea_9bb5_11d095fcd02c.slice/crio-f373fdb9729322bdb9ea9ed5767b278c5150533829f3d50d51b3e71e3fb68a86 WatchSource:0}: Error finding container f373fdb9729322bdb9ea9ed5767b278c5150533829f3d50d51b3e71e3fb68a86: Status 404 returned error can't find the container with id f373fdb9729322bdb9ea9ed5767b278c5150533829f3d50d51b3e71e3fb68a86
Oct 30 13:40:55.310692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:55.310626    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9" event={"ID":"8413eea7-dfb7-44ea-9bb5-11d095fcd02c","Type":"ContainerStarted","Data":"f2eacefd9437da25b2132ab13a0c67474e4bc54c60753883240bc74a19707c86"}
Oct 30 13:40:55.311248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:55.310720    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9" event={"ID":"8413eea7-dfb7-44ea-9bb5-11d095fcd02c","Type":"ContainerStarted","Data":"f373fdb9729322bdb9ea9ed5767b278c5150533829f3d50d51b3e71e3fb68a86"}
Oct 30 13:40:55.311248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:55.310750    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:40:55.352056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:55.351998    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9" podStartSLOduration=2.351982704 podStartE2EDuration="2.351982704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:40:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:40:55.347911046 +0000 UTC m=+2265.608149293" watchObservedRunningTime="2026-10-30 13:40:55.351982704 +0000 UTC m=+2265.612220994"
Oct 30 13:40:55.892570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:40:55.892524    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4edcd9f1-9bea-456a-b298-828f5b727341" path="/var/lib/kubelet/pods/4edcd9f1-9bea-456a-b298-828f5b727341/volumes"
Oct 30 13:41:01.191289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.191244    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"]
Oct 30 13:41:01.192733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.192705    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4edcd9f1-9bea-456a-b298-828f5b727341" containerName="payload-processing"
Oct 30 13:41:01.192733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.192733    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4edcd9f1-9bea-456a-b298-828f5b727341" containerName="payload-processing"
Oct 30 13:41:01.193115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.193099    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4edcd9f1-9bea-456a-b298-828f5b727341" containerName="payload-processing"
Oct 30 13:41:01.198286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.198263    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.208754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.208728    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"]
Oct 30 13:41:01.220506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.216824    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:41:01.220506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.216838    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:41:01.220506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.216919    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:41:01.220506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.216951    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:41:01.258569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.258537    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.258759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.258589    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.258816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.258757    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5jdt\" (UniqueName: \"kubernetes.io/projected/20d96928-a60b-41b8-be36-56fe6a1e8d4e-kube-api-access-b5jdt\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.266348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.266318    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:41:01.360381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.360340    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.360577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.360431    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.360670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.360630    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5jdt\" (UniqueName: \"kubernetes.io/projected/20d96928-a60b-41b8-be36-56fe6a1e8d4e-kube-api-access-b5jdt\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.362412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.362377    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.364453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.364425    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.379495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.379458    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5jdt\" (UniqueName: \"kubernetes.io/projected/20d96928-a60b-41b8-be36-56fe6a1e8d4e-kube-api-access-b5jdt\") pod \"maas-api-revoke-sub-56f1414f-tdzxk\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.515172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.515078    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:01.824490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:01.824460    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"]
Oct 30 13:41:01.826196 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:41:01.826172    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20d96928_a60b_41b8_be36_56fe6a1e8d4e.slice/crio-88ab543c292700f3115cc3a9f22952123c9b18a0896dfb0abaf0a4badc5db494 WatchSource:0}: Error finding container 88ab543c292700f3115cc3a9f22952123c9b18a0896dfb0abaf0a4badc5db494: Status 404 returned error can't find the container with id 88ab543c292700f3115cc3a9f22952123c9b18a0896dfb0abaf0a4badc5db494
Oct 30 13:41:02.371573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:02.371529    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk" event={"ID":"20d96928-a60b-41b8-be36-56fe6a1e8d4e","Type":"ContainerStarted","Data":"bdc642dcd79fc9a4a56511dc44e4963e361930a3c08d414e7035295f762a94e7"}
Oct 30 13:41:02.372094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:02.371606    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk" event={"ID":"20d96928-a60b-41b8-be36-56fe6a1e8d4e","Type":"ContainerStarted","Data":"88ab543c292700f3115cc3a9f22952123c9b18a0896dfb0abaf0a4badc5db494"}
Oct 30 13:41:02.412975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:02.412742    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk" podStartSLOduration=1.412724861 podStartE2EDuration="1.412724861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:02.408258373 +0000 UTC m=+2272.668496618" watchObservedRunningTime="2026-10-30 13:41:02.412724861 +0000 UTC m=+2272.672963096"
Oct 30 13:41:03.380950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:03.380916    4291 generic.go:356] "Generic (PLEG): container finished" podID="20d96928-a60b-41b8-be36-56fe6a1e8d4e" containerID="bdc642dcd79fc9a4a56511dc44e4963e361930a3c08d414e7035295f762a94e7" exitCode=0
Oct 30 13:41:03.381367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:03.381003    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk" event={"ID":"20d96928-a60b-41b8-be36-56fe6a1e8d4e","Type":"ContainerDied","Data":"bdc642dcd79fc9a4a56511dc44e4963e361930a3c08d414e7035295f762a94e7"}
Oct 30 13:41:04.586415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.586382    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:04.710080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.709988    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-xks-ca\") pod \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") "
Oct 30 13:41:04.710080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.710030    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20d96928-a60b-41b8-be36-56fe6a1e8d4e-kube-api-access-b5jdt\" (UniqueName: \"kubernetes.io/projected/20d96928-a60b-41b8-be36-56fe6a1e8d4e-kube-api-access-b5jdt\") pod \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") "
Oct 30 13:41:04.710337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.710105    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-openshift-ca\") pod \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\" (UID: \"20d96928-a60b-41b8-be36-56fe6a1e8d4e\") "
Oct 30 13:41:04.710515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.710477    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-openshift-ca" pod "20d96928-a60b-41b8-be36-56fe6a1e8d4e" (UID: "20d96928-a60b-41b8-be36-56fe6a1e8d4e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:04.710720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.710706    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.712222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.712199    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-xks-ca" pod "20d96928-a60b-41b8-be36-56fe6a1e8d4e" (UID: "20d96928-a60b-41b8-be36-56fe6a1e8d4e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:04.712292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.712272    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20d96928-a60b-41b8-be36-56fe6a1e8d4e-kube-api-access-b5jdt" pod "20d96928-a60b-41b8-be36-56fe6a1e8d4e" (UID: "20d96928-a60b-41b8-be36-56fe6a1e8d4e"). InnerVolumeSpecName "kube-api-access-b5jdt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:04.812266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.812232    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20d96928-a60b-41b8-be36-56fe6a1e8d4e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:04.812266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:04.812260    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5jdt\" (UniqueName: \"kubernetes.io/projected/20d96928-a60b-41b8-be36-56fe6a1e8d4e-kube-api-access-b5jdt\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:05.400498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:05.400461    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"
Oct 30 13:41:05.400705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:05.400493    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk" event={"ID":"20d96928-a60b-41b8-be36-56fe6a1e8d4e","Type":"ContainerDied","Data":"88ab543c292700f3115cc3a9f22952123c9b18a0896dfb0abaf0a4badc5db494"}
Oct 30 13:41:05.400705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:05.400553    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ab543c292700f3115cc3a9f22952123c9b18a0896dfb0abaf0a4badc5db494"
Oct 30 13:41:06.321847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:06.321815    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:41:06.392018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:06.391984    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"]
Oct 30 13:41:08.296439 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:41:08.296404    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:41:11.393342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.393287    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w" podUID="c40fa1e4-849d-4660-81f6-832b74efc24e" containerName="payload-processing" containerID="cri-o://10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592" gracePeriod=25
Oct 30 13:41:11.718227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.718200    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:41:11.834358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.834320    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-provider-credentials\") pod \"c40fa1e4-849d-4660-81f6-832b74efc24e\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") "
Oct 30 13:41:11.834566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.834434    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-routing-overlay-volume\") pod \"c40fa1e4-849d-4660-81f6-832b74efc24e\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") "
Oct 30 13:41:11.834566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.834528    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-kube-api-access-9gk6w\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-kube-api-access-9gk6w\") pod \"c40fa1e4-849d-4660-81f6-832b74efc24e\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") "
Oct 30 13:41:11.834566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.834558    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-plugins-config-volume\") pod \"c40fa1e4-849d-4660-81f6-832b74efc24e\" (UID: \"c40fa1e4-849d-4660-81f6-832b74efc24e\") "
Oct 30 13:41:11.835457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.835430    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-routing-overlay-volume" pod "c40fa1e4-849d-4660-81f6-832b74efc24e" (UID: "c40fa1e4-849d-4660-81f6-832b74efc24e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:11.835587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.835526    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-plugins-config-volume" pod "c40fa1e4-849d-4660-81f6-832b74efc24e" (UID: "c40fa1e4-849d-4660-81f6-832b74efc24e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:11.837459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.837430    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-kube-api-access-9gk6w" pod "c40fa1e4-849d-4660-81f6-832b74efc24e" (UID: "c40fa1e4-849d-4660-81f6-832b74efc24e"). InnerVolumeSpecName "kube-api-access-9gk6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:11.837557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.837516    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-provider-credentials" pod "c40fa1e4-849d-4660-81f6-832b74efc24e" (UID: "c40fa1e4-849d-4660-81f6-832b74efc24e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:11.838486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.838465    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"]
Oct 30 13:41:11.839451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.839433    4291 cpu_manager.go:414] "containerMap: removing container" podUID="20d96928-a60b-41b8-be36-56fe6a1e8d4e" containerName="revoke-keys"
Oct 30 13:41:11.839504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.839455    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20d96928-a60b-41b8-be36-56fe6a1e8d4e" containerName="revoke-keys"
Oct 30 13:41:11.839539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.839504    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c40fa1e4-849d-4660-81f6-832b74efc24e" containerName="payload-processing"
Oct 30 13:41:11.839539 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.839513    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c40fa1e4-849d-4660-81f6-832b74efc24e" containerName="payload-processing"
Oct 30 13:41:11.839706 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.839692    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="20d96928-a60b-41b8-be36-56fe6a1e8d4e" containerName="revoke-keys"
Oct 30 13:41:11.839747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.839715    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c40fa1e4-849d-4660-81f6-832b74efc24e" containerName="payload-processing"
Oct 30 13:41:11.844057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.844038    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:11.863517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.860937    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:41:11.863517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.861679    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:41:11.863517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.861754    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:41:11.863517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.861840    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:41:11.937934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.937357    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be0a024-6330-453c-b905-8c6094669b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:11.937934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.937538    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtmm\" (UniqueName: \"kubernetes.io/projected/0be0a024-6330-453c-b905-8c6094669b07-kube-api-access-nwtmm\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:11.937934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.937653    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be0a024-6330-453c-b905-8c6094669b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:11.937934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.937733    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:11.937934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.937750    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gk6w\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-kube-api-access-9gk6w\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:11.937934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.937765    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c40fa1e4-849d-4660-81f6-832b74efc24e-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:11.937934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.937777    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c40fa1e4-849d-4660-81f6-832b74efc24e-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:11.980659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:11.980413    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"]
Oct 30 13:41:12.038906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.038863    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be0a024-6330-453c-b905-8c6094669b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:12.039123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.038978    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be0a024-6330-453c-b905-8c6094669b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:12.039190 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.039157    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtmm\" (UniqueName: \"kubernetes.io/projected/0be0a024-6330-453c-b905-8c6094669b07-kube-api-access-nwtmm\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:12.039792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.039761    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be0a024-6330-453c-b905-8c6094669b07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:12.041664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.041609    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be0a024-6330-453c-b905-8c6094669b07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:12.084808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.084396    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtmm\" (UniqueName: \"kubernetes.io/projected/0be0a024-6330-453c-b905-8c6094669b07-kube-api-access-nwtmm\") pod \"maas-api-revoke-sub-1eb69cfc-69lwr\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:12.313727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.312688    4291 scope.go:122] "RemoveContainer" containerID="28b8a3f045c9ff1d2af4c8390cb83c324f592840fc7a8b7a55e9f0e922640e24"
Oct 30 13:41:12.313727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.313449    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:12.413349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.412424    4291 scope.go:122] "RemoveContainer" containerID="3e05b00790031ba12b29caa2a5c803c400c3eef9f5b7b0a465b1ae4c870629aa"
Oct 30 13:41:12.444985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.444874    4291 scope.go:122] "RemoveContainer" containerID="2f20cd45ad032c4aa37fed0109f2e42d31e3f0ad0aa7bb9b45d40c9299120067"
Oct 30 13:41:12.520478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.520447    4291 generic.go:356] "Generic (PLEG): container finished" podID="c40fa1e4-849d-4660-81f6-832b74efc24e" containerID="10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592" exitCode=0
Oct 30 13:41:12.522056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.520706    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"
Oct 30 13:41:12.522056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.520733    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w" event={"ID":"c40fa1e4-849d-4660-81f6-832b74efc24e","Type":"ContainerDied","Data":"10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592"}
Oct 30 13:41:12.522056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.521113    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w" event={"ID":"c40fa1e4-849d-4660-81f6-832b74efc24e","Type":"ContainerDied","Data":"38d0374ba432bf8d272aafa9ec7e91a66a6a5360c10b78a738fecb0519bc8f0d"}
Oct 30 13:41:12.522056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.521153    4291 scope.go:122] "RemoveContainer" containerID="10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592"
Oct 30 13:41:12.548336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.548315    4291 scope.go:122] "RemoveContainer" containerID="10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592"
Oct 30 13:41:12.548753 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:41:12.548726    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592\": container with ID starting with 10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592 not found: ID does not exist" containerID="10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592"
Oct 30 13:41:12.548824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.548769    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592"} err="failed to get container status \"10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592\": rpc error: code = NotFound desc = could not find container \"10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592\": container with ID starting with 10260047f74299b127b3e9b83e25a3768a538f59f25f1bbf79277d8cdf037592 not found: ID does not exist"
Oct 30 13:41:12.583813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.583722    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"]
Oct 30 13:41:12.587388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.587363    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-647cn8h2w"]
Oct 30 13:41:12.737939 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:12.737915    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"]
Oct 30 13:41:12.739122 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:41:12.739098    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0be0a024_6330_453c_b905_8c6094669b07.slice/crio-6c416a711cab5784a9feaf77a0cb02fe61430cc97c7705bc036909e519620ee7 WatchSource:0}: Error finding container 6c416a711cab5784a9feaf77a0cb02fe61430cc97c7705bc036909e519620ee7: Status 404 returned error can't find the container with id 6c416a711cab5784a9feaf77a0cb02fe61430cc97c7705bc036909e519620ee7
Oct 30 13:41:13.564316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:13.564168    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr" event={"ID":"0be0a024-6330-453c-b905-8c6094669b07","Type":"ContainerStarted","Data":"b474eae83c75317ff8e25e4aa08f93f4019270208faa6a0bd6daa215e4499030"}
Oct 30 13:41:13.564316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:13.564278    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr" event={"ID":"0be0a024-6330-453c-b905-8c6094669b07","Type":"ContainerStarted","Data":"6c416a711cab5784a9feaf77a0cb02fe61430cc97c7705bc036909e519620ee7"}
Oct 30 13:41:13.604178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:13.604125    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr" podStartSLOduration=2.604110202 podStartE2EDuration="2.604110202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:13.602426873 +0000 UTC m=+2283.862665107" watchObservedRunningTime="2026-10-30 13:41:13.604110202 +0000 UTC m=+2283.864348436"
Oct 30 13:41:13.884879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:13.884847    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c40fa1e4-849d-4660-81f6-832b74efc24e" path="/var/lib/kubelet/pods/c40fa1e4-849d-4660-81f6-832b74efc24e/volumes"
Oct 30 13:41:14.572778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:14.572745    4291 generic.go:356] "Generic (PLEG): container finished" podID="0be0a024-6330-453c-b905-8c6094669b07" containerID="b474eae83c75317ff8e25e4aa08f93f4019270208faa6a0bd6daa215e4499030" exitCode=0
Oct 30 13:41:14.573186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:14.572843    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr" event={"ID":"0be0a024-6330-453c-b905-8c6094669b07","Type":"ContainerDied","Data":"b474eae83c75317ff8e25e4aa08f93f4019270208faa6a0bd6daa215e4499030"}
Oct 30 13:41:15.768841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.768813    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:15.881278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.881241    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0be0a024-6330-453c-b905-8c6094669b07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be0a024-6330-453c-b905-8c6094669b07-maas-api-openshift-ca\") pod \"0be0a024-6330-453c-b905-8c6094669b07\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") "
Oct 30 13:41:15.881495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.881349    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0be0a024-6330-453c-b905-8c6094669b07-kube-api-access-nwtmm\" (UniqueName: \"kubernetes.io/projected/0be0a024-6330-453c-b905-8c6094669b07-kube-api-access-nwtmm\") pod \"0be0a024-6330-453c-b905-8c6094669b07\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") "
Oct 30 13:41:15.881495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.881473    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0be0a024-6330-453c-b905-8c6094669b07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be0a024-6330-453c-b905-8c6094669b07-maas-api-xks-ca\") pod \"0be0a024-6330-453c-b905-8c6094669b07\" (UID: \"0be0a024-6330-453c-b905-8c6094669b07\") "
Oct 30 13:41:15.881983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.881950    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0be0a024-6330-453c-b905-8c6094669b07-maas-api-openshift-ca" pod "0be0a024-6330-453c-b905-8c6094669b07" (UID: "0be0a024-6330-453c-b905-8c6094669b07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:15.884402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.884356    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0be0a024-6330-453c-b905-8c6094669b07-maas-api-xks-ca" pod "0be0a024-6330-453c-b905-8c6094669b07" (UID: "0be0a024-6330-453c-b905-8c6094669b07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:15.884986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.884961    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0be0a024-6330-453c-b905-8c6094669b07-kube-api-access-nwtmm" pod "0be0a024-6330-453c-b905-8c6094669b07" (UID: "0be0a024-6330-453c-b905-8c6094669b07"). InnerVolumeSpecName "kube-api-access-nwtmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:15.983476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.983414    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtmm\" (UniqueName: \"kubernetes.io/projected/0be0a024-6330-453c-b905-8c6094669b07-kube-api-access-nwtmm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:15.983703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.983504    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0be0a024-6330-453c-b905-8c6094669b07-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:15.983703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:15.983566    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0be0a024-6330-453c-b905-8c6094669b07-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:16.590745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:16.590713    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"
Oct 30 13:41:16.590745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:16.590733    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr" event={"ID":"0be0a024-6330-453c-b905-8c6094669b07","Type":"ContainerDied","Data":"6c416a711cab5784a9feaf77a0cb02fe61430cc97c7705bc036909e519620ee7"}
Oct 30 13:41:16.590934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:16.590794    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c416a711cab5784a9feaf77a0cb02fe61430cc97c7705bc036909e519620ee7"
Oct 30 13:41:18.621229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:18.621191    4291 generic.go:356] "Generic (PLEG): container finished" podID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerID="a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6" exitCode=0
Oct 30 13:41:18.621825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:18.621260    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" event={"ID":"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b","Type":"ContainerDied","Data":"a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6"}
Oct 30 13:41:18.621825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:18.621743    4291 scope.go:122] "RemoveContainer" containerID="a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6"
Oct 30 13:41:18.623768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:18.623737    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:41:18.623867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:18.623834    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:41:18.623913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:18.623882    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:41:19.632698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:19.632665    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" event={"ID":"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b","Type":"ContainerStarted","Data":"892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a"}
Oct 30 13:41:20.101402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:20.101360    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:41:20.101590 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:20.101570    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:41:20.101668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:20.101619    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:41:21.108661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:21.104079    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:41:21.108661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:21.104146    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:41:22.101803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:22.101763    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:41:22.101995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:22.101839    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:41:23.031837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:23.031733    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"]
Oct 30 13:41:23.043369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:23.043330    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68a86172-zjjms"]
Oct 30 13:41:23.105209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:23.104890    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:41:23.105535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:23.105500    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:41:23.884016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:23.883981    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d6b581a-3428-45e9-aa28-76b5e6aa4aac" path="/var/lib/kubelet/pods/6d6b581a-3428-45e9-aa28-76b5e6aa4aac/volumes"
Oct 30 13:41:24.101413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.101367    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:41:24.102147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.101444    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:41:24.280853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.280748    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"]
Oct 30 13:41:24.282141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.282109    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0be0a024-6330-453c-b905-8c6094669b07" containerName="revoke-keys"
Oct 30 13:41:24.282339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.282324    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0be0a024-6330-453c-b905-8c6094669b07" containerName="revoke-keys"
Oct 30 13:41:24.282807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.282785    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0be0a024-6330-453c-b905-8c6094669b07" containerName="revoke-keys"
Oct 30 13:41:24.323703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.323667    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"]
Oct 30 13:41:24.324031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.324009    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.340653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.339714    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:41:24.340653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.339873    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:41:24.340653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.339979    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:41:24.340653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.340071    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:41:24.346406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.346370    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.346598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.346431    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.346702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.346602    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv4cc\" (UniqueName: \"kubernetes.io/projected/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-kube-api-access-fv4cc\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.447523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.447481    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.447771 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.447557    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.447771 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.447753    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv4cc\" (UniqueName: \"kubernetes.io/projected/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-kube-api-access-fv4cc\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.448307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.448285    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.477662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.477001    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.480401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.480311    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv4cc\" (UniqueName: \"kubernetes.io/projected/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-kube-api-access-fv4cc\") pod \"maas-api-revoke-sub-e1668c6f-jc5fd\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.644792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.644752    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:24.894359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.894319    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"]
Oct 30 13:41:24.983714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:24.983680    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"]
Oct 30 13:41:24.984251 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:41:24.984228    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdcc5d2d_1d9f_4d8f_8b87_dd43a5953ea1.slice/crio-a47729976183864e509d19f21887dd97b24e48d900fd222e4b3d9251ec1c6932 WatchSource:0}: Error finding container a47729976183864e509d19f21887dd97b24e48d900fd222e4b3d9251ec1c6932: Status 404 returned error can't find the container with id a47729976183864e509d19f21887dd97b24e48d900fd222e4b3d9251ec1c6932
Oct 30 13:41:25.101953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:25.101915    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused" start-of-body=
Oct 30 13:41:25.102353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:25.101970    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.209:15021/healthz/ready\": dial tcp 10.133.0.209:15021: connect: connection refused"
Oct 30 13:41:25.712905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:25.712804    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd" event={"ID":"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1","Type":"ContainerStarted","Data":"60ecec4c121157fd9176bc157d020d4d41da2617a928ea62774fd2f67b07373b"}
Oct 30 13:41:25.712905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:25.712863    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd" event={"ID":"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1","Type":"ContainerStarted","Data":"a47729976183864e509d19f21887dd97b24e48d900fd222e4b3d9251ec1c6932"}
Oct 30 13:41:25.754674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:25.754570    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd" podStartSLOduration=1.754550214 podStartE2EDuration="1.754550214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:25.74961901 +0000 UTC m=+2296.009857257" watchObservedRunningTime="2026-10-30 13:41:25.754550214 +0000 UTC m=+2296.014788450"
Oct 30 13:41:26.106993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:26.106960    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:41:26.107513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:26.107389    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:41:26.108159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:26.108137    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 13:41:26.721286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:26.721256    4291 generic.go:356] "Generic (PLEG): container finished" podID="fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" containerID="60ecec4c121157fd9176bc157d020d4d41da2617a928ea62774fd2f67b07373b" exitCode=0
Oct 30 13:41:26.721456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:26.721338    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd" event={"ID":"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1","Type":"ContainerDied","Data":"60ecec4c121157fd9176bc157d020d4d41da2617a928ea62774fd2f67b07373b"}
Oct 30 13:41:27.891397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.891373    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:27.922069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.922041    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-kube-api-access-fv4cc\" (UniqueName: \"kubernetes.io/projected/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-kube-api-access-fv4cc\") pod \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") "
Oct 30 13:41:27.922252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.922233    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-xks-ca\") pod \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") "
Oct 30 13:41:27.922342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.922328    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-openshift-ca\") pod \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\" (UID: \"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1\") "
Oct 30 13:41:27.922688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.922661    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-openshift-ca" pod "fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" (UID: "fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:27.922903 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.922885    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:27.924714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.924694    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-xks-ca" pod "fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" (UID: "fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:27.924714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:27.924702    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-kube-api-access-fv4cc" pod "fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" (UID: "fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1"). InnerVolumeSpecName "kube-api-access-fv4cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:28.024833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:28.024730    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:28.024833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:28.024775    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv4cc\" (UniqueName: \"kubernetes.io/projected/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1-kube-api-access-fv4cc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:28.736217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:28.736187    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"
Oct 30 13:41:28.736409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:28.736221    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd" event={"ID":"fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1","Type":"ContainerDied","Data":"a47729976183864e509d19f21887dd97b24e48d900fd222e4b3d9251ec1c6932"}
Oct 30 13:41:28.736409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:28.736297    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a47729976183864e509d19f21887dd97b24e48d900fd222e4b3d9251ec1c6932"
Oct 30 13:41:29.895054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:29.895005    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz" podUID="b6848e19-f9e3-4282-a19a-2262d9b72d17" containerName="payload-processing" containerID="cri-o://e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5" gracePeriod=25
Oct 30 13:41:30.180840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.180814    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:41:30.254985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.254943    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-plugins-config-volume\") pod \"b6848e19-f9e3-4282-a19a-2262d9b72d17\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") "
Oct 30 13:41:30.255195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.255064    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-routing-overlay-volume\") pod \"b6848e19-f9e3-4282-a19a-2262d9b72d17\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") "
Oct 30 13:41:30.255195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.255126    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-provider-credentials\") pod \"b6848e19-f9e3-4282-a19a-2262d9b72d17\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") "
Oct 30 13:41:30.255195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.255159    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-kube-api-access-s9j5l\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-kube-api-access-s9j5l\") pod \"b6848e19-f9e3-4282-a19a-2262d9b72d17\" (UID: \"b6848e19-f9e3-4282-a19a-2262d9b72d17\") "
Oct 30 13:41:30.255492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.255459    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-plugins-config-volume" pod "b6848e19-f9e3-4282-a19a-2262d9b72d17" (UID: "b6848e19-f9e3-4282-a19a-2262d9b72d17"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:30.255571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.255511    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-routing-overlay-volume" pod "b6848e19-f9e3-4282-a19a-2262d9b72d17" (UID: "b6848e19-f9e3-4282-a19a-2262d9b72d17"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:30.255921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.255896    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.256034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.255925    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b6848e19-f9e3-4282-a19a-2262d9b72d17-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.257922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.257897    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-kube-api-access-s9j5l" pod "b6848e19-f9e3-4282-a19a-2262d9b72d17" (UID: "b6848e19-f9e3-4282-a19a-2262d9b72d17"). InnerVolumeSpecName "kube-api-access-s9j5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:30.258027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.257936    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-provider-credentials" pod "b6848e19-f9e3-4282-a19a-2262d9b72d17" (UID: "b6848e19-f9e3-4282-a19a-2262d9b72d17"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:30.357044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.357011    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9j5l\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-kube-api-access-s9j5l\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.357044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.357039    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b6848e19-f9e3-4282-a19a-2262d9b72d17-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:30.750813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.750773    4291 generic.go:356] "Generic (PLEG): container finished" podID="b6848e19-f9e3-4282-a19a-2262d9b72d17" containerID="e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5" exitCode=0
Oct 30 13:41:30.751031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.750830    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"
Oct 30 13:41:30.751031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.750882    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz" event={"ID":"b6848e19-f9e3-4282-a19a-2262d9b72d17","Type":"ContainerDied","Data":"e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5"}
Oct 30 13:41:30.751031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.750975    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz" event={"ID":"b6848e19-f9e3-4282-a19a-2262d9b72d17","Type":"ContainerDied","Data":"155adf4862267a24980e2d3f4d5b7577c5ae85f97844aadea39cf5f3a001c398"}
Oct 30 13:41:30.751031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.751029    4291 scope.go:122] "RemoveContainer" containerID="e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5"
Oct 30 13:41:30.761971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.761876    4291 scope.go:122] "RemoveContainer" containerID="e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5"
Oct 30 13:41:30.763406 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:41:30.763302    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5\": container with ID starting with e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5 not found: ID does not exist" containerID="e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5"
Oct 30 13:41:30.763406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.763340    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5"} err="failed to get container status \"e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5\": rpc error: code = NotFound desc = could not find container \"e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5\": container with ID starting with e037a104dc183e68df9f89b3b748458be169733eb2c91afa6708513848b2adc5 not found: ID does not exist"
Oct 30 13:41:30.828497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.828459    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"]
Oct 30 13:41:30.831856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:30.831829    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-5c68mbdbz"]
Oct 30 13:41:31.881206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:31.881171    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6848e19-f9e3-4282-a19a-2262d9b72d17" path="/var/lib/kubelet/pods/b6848e19-f9e3-4282-a19a-2262d9b72d17/volumes"
Oct 30 13:41:33.035061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:33.035028    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"]
Oct 30 13:41:33.046893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:33.046860    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-923e5b73-k8q8h"]
Oct 30 13:41:33.880177 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:33.880142    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd3f4b78-9725-49e5-9427-0a1f974e898a" path="/var/lib/kubelet/pods/fd3f4b78-9725-49e5-9427-0a1f974e898a/volumes"
Oct 30 13:41:35.488169 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:35.488133    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"]
Oct 30 13:41:37.461024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.460989    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"]
Oct 30 13:41:37.461656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.461625    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" containerName="revoke-keys"
Oct 30 13:41:37.461656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.461656    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" containerName="revoke-keys"
Oct 30 13:41:37.461768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.461671    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b6848e19-f9e3-4282-a19a-2262d9b72d17" containerName="payload-processing"
Oct 30 13:41:37.461768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.461676    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6848e19-f9e3-4282-a19a-2262d9b72d17" containerName="payload-processing"
Oct 30 13:41:37.461836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.461800    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" containerName="revoke-keys"
Oct 30 13:41:37.461836 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.461810    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6848e19-f9e3-4282-a19a-2262d9b72d17" containerName="payload-processing"
Oct 30 13:41:37.465334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.465317    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.483405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.483369    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"]
Oct 30 13:41:37.491356 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.491324    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"kube-root-ca.crt\""
Oct 30 13:41:37.491740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.491717    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-48f199422eae\""
Oct 30 13:41:37.492067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.492048    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-e2e--cb40fe81-dockercfg-5rb7k\""
Oct 30 13:41:37.492189 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.492126    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"routing-overlay\""
Oct 30 13:41:37.492262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.492223    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"openshift-service-ca.crt\""
Oct 30 13:41:37.492325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.492257    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d2cecb\""
Oct 30 13:41:37.589219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.589164    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.589394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.589247    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.589465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.589383    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.589465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.589449    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng7sh\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-kube-api-access-ng7sh\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.691021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.690977    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.691243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.691031    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.691243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.691116    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.691243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.691173    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng7sh\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-kube-api-access-ng7sh\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.691881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.691852    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.692003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.691899    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.693623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.693592    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.710217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.710184    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng7sh\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-kube-api-access-ng7sh\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:37.779523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:37.779426    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:38.115652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:38.115612    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"]
Oct 30 13:41:38.116410 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:41:38.116384    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d531323_5434_472e_a5c3_5a40238f0ee2.slice/crio-7c62109c73b24765ca5d66471c05aeac59b8f0ca0d8abf987af3db2ff3f70497 WatchSource:0}: Error finding container 7c62109c73b24765ca5d66471c05aeac59b8f0ca0d8abf987af3db2ff3f70497: Status 404 returned error can't find the container with id 7c62109c73b24765ca5d66471c05aeac59b8f0ca0d8abf987af3db2ff3f70497
Oct 30 13:41:38.211220 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:41:38.211187    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:41:38.813705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:38.813662    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm" event={"ID":"9d531323-5434-472e-a5c3-5a40238f0ee2","Type":"ContainerStarted","Data":"d8ee8ea382fd038e0b0be59a6af545aaebe33fb38ace4c893d1e619eb0d08aa3"}
Oct 30 13:41:38.814144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:38.813731    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm" event={"ID":"9d531323-5434-472e-a5c3-5a40238f0ee2","Type":"ContainerStarted","Data":"7c62109c73b24765ca5d66471c05aeac59b8f0ca0d8abf987af3db2ff3f70497"}
Oct 30 13:41:38.814144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:38.813861    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:38.853782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:38.853725    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm" podStartSLOduration=1.853711516 podStartE2EDuration="1.853711516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:38.852460516 +0000 UTC m=+2309.112698751" watchObservedRunningTime="2026-10-30 13:41:38.853711516 +0000 UTC m=+2309.113949750"
Oct 30 13:41:39.365538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:39.365505    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:41:40.488913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:40.488865    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9" podUID="8413eea7-dfb7-44ea-9bb5-11d095fcd02c" containerName="payload-processing" containerID="cri-o://f2eacefd9437da25b2132ab13a0c67474e4bc54c60753883240bc74a19707c86" gracePeriod=25
Oct 30 13:41:40.878659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:40.878025    4291 generic.go:356] "Generic (PLEG): container finished" podID="8413eea7-dfb7-44ea-9bb5-11d095fcd02c" containerID="f2eacefd9437da25b2132ab13a0c67474e4bc54c60753883240bc74a19707c86" exitCode=0
Oct 30 13:41:40.878659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:40.878146    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9" event={"ID":"8413eea7-dfb7-44ea-9bb5-11d095fcd02c","Type":"ContainerDied","Data":"f2eacefd9437da25b2132ab13a0c67474e4bc54c60753883240bc74a19707c86"}
Oct 30 13:41:41.114367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.114336    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:41:41.280385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.280283    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-plugins-config-volume\") pod \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") "
Oct 30 13:41:41.280385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.280376    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-kube-api-access-dh6st\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-kube-api-access-dh6st\") pod \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") "
Oct 30 13:41:41.280692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.280526    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-provider-credentials\") pod \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") "
Oct 30 13:41:41.280692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.280681    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-routing-overlay-volume\") pod \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\" (UID: \"8413eea7-dfb7-44ea-9bb5-11d095fcd02c\") "
Oct 30 13:41:41.281014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.280968    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-plugins-config-volume" pod "8413eea7-dfb7-44ea-9bb5-11d095fcd02c" (UID: "8413eea7-dfb7-44ea-9bb5-11d095fcd02c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:41.281376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.281352    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-routing-overlay-volume" pod "8413eea7-dfb7-44ea-9bb5-11d095fcd02c" (UID: "8413eea7-dfb7-44ea-9bb5-11d095fcd02c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:41.281465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.281444    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:41.282960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.282932    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-kube-api-access-dh6st" pod "8413eea7-dfb7-44ea-9bb5-11d095fcd02c" (UID: "8413eea7-dfb7-44ea-9bb5-11d095fcd02c"). InnerVolumeSpecName "kube-api-access-dh6st". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:41.284057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.284031    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-provider-credentials" pod "8413eea7-dfb7-44ea-9bb5-11d095fcd02c" (UID: "8413eea7-dfb7-44ea-9bb5-11d095fcd02c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:41.382564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.382526    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:41.382564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.382559    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:41.382564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.382572    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dh6st\" (UniqueName: \"kubernetes.io/projected/8413eea7-dfb7-44ea-9bb5-11d095fcd02c-kube-api-access-dh6st\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:41.887618 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.887537    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9" event={"ID":"8413eea7-dfb7-44ea-9bb5-11d095fcd02c","Type":"ContainerDied","Data":"f373fdb9729322bdb9ea9ed5767b278c5150533829f3d50d51b3e71e3fb68a86"}
Oct 30 13:41:41.888200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.887591    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"
Oct 30 13:41:41.888200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.887666    4291 scope.go:122] "RemoveContainer" containerID="f2eacefd9437da25b2132ab13a0c67474e4bc54c60753883240bc74a19707c86"
Oct 30 13:41:41.938673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.938625    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"]
Oct 30 13:41:41.950503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:41.950465    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-844856/payload-processing-external-model-e2e-extmod-b-844856-57cf7dpg9"]
Oct 30 13:41:43.550716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.550677    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"]
Oct 30 13:41:43.551305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.551291    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8413eea7-dfb7-44ea-9bb5-11d095fcd02c" containerName="payload-processing"
Oct 30 13:41:43.551355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.551306    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8413eea7-dfb7-44ea-9bb5-11d095fcd02c" containerName="payload-processing"
Oct 30 13:41:43.551415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.551405    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8413eea7-dfb7-44ea-9bb5-11d095fcd02c" containerName="payload-processing"
Oct 30 13:41:43.556397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.556376    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.573090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.573057    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"]
Oct 30 13:41:43.581286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.581241    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:41:43.581286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.581245    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:41:43.581491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.581297    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:41:43.588957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.588927    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:41:43.615697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.615621    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.615919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.615737    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps5b6\" (UniqueName: \"kubernetes.io/projected/4ae0e243-00fc-4b45-85e9-52d527e5fb25-kube-api-access-ps5b6\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.616006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.615950    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.716702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.716670    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.716908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.716723    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps5b6\" (UniqueName: \"kubernetes.io/projected/4ae0e243-00fc-4b45-85e9-52d527e5fb25-kube-api-access-ps5b6\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.716908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.716866    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.717503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.717482    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.719020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.719003    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.736382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.736347    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps5b6\" (UniqueName: \"kubernetes.io/projected/4ae0e243-00fc-4b45-85e9-52d527e5fb25-kube-api-access-ps5b6\") pod \"maas-api-revoke-sub-cf7a0fe6-77rmq\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.871206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.871167    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:43.882440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:43.882408    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8413eea7-dfb7-44ea-9bb5-11d095fcd02c" path="/var/lib/kubelet/pods/8413eea7-dfb7-44ea-9bb5-11d095fcd02c/volumes"
Oct 30 13:41:44.181287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:44.181258    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"]
Oct 30 13:41:44.182906 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:41:44.182877    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae0e243_00fc_4b45_85e9_52d527e5fb25.slice/crio-fadd0f88677c1e88d39de91a28c831333eaeca055ac675748aecbdd24151f80b WatchSource:0}: Error finding container fadd0f88677c1e88d39de91a28c831333eaeca055ac675748aecbdd24151f80b: Status 404 returned error can't find the container with id fadd0f88677c1e88d39de91a28c831333eaeca055ac675748aecbdd24151f80b
Oct 30 13:41:44.925023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:44.924982    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq" event={"ID":"4ae0e243-00fc-4b45-85e9-52d527e5fb25","Type":"ContainerStarted","Data":"663d1c1bc427c3f7af3d59f000b9ecaadfda862fe9d434fb2ab5eaf4e99718e7"}
Oct 30 13:41:44.925461 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:44.925080    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq" event={"ID":"4ae0e243-00fc-4b45-85e9-52d527e5fb25","Type":"ContainerStarted","Data":"fadd0f88677c1e88d39de91a28c831333eaeca055ac675748aecbdd24151f80b"}
Oct 30 13:41:45.970506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:45.970470    4291 generic.go:356] "Generic (PLEG): container finished" podID="4ae0e243-00fc-4b45-85e9-52d527e5fb25" containerID="663d1c1bc427c3f7af3d59f000b9ecaadfda862fe9d434fb2ab5eaf4e99718e7" exitCode=0
Oct 30 13:41:45.971301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:45.971276    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq" event={"ID":"4ae0e243-00fc-4b45-85e9-52d527e5fb25","Type":"ContainerDied","Data":"663d1c1bc427c3f7af3d59f000b9ecaadfda862fe9d434fb2ab5eaf4e99718e7"}
Oct 30 13:41:46.896191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:46.896151    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"]
Oct 30 13:41:46.899086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:46.899058    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:47.151494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.151418    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:47.224101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.224064    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-openshift-ca\") pod \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") "
Oct 30 13:41:47.224324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.224125    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ae0e243-00fc-4b45-85e9-52d527e5fb25-kube-api-access-ps5b6\" (UniqueName: \"kubernetes.io/projected/4ae0e243-00fc-4b45-85e9-52d527e5fb25-kube-api-access-ps5b6\") pod \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") "
Oct 30 13:41:47.224324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.224231    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-xks-ca\") pod \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\" (UID: \"4ae0e243-00fc-4b45-85e9-52d527e5fb25\") "
Oct 30 13:41:47.224531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.224508    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-openshift-ca" pod "4ae0e243-00fc-4b45-85e9-52d527e5fb25" (UID: "4ae0e243-00fc-4b45-85e9-52d527e5fb25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:47.224799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.224779    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:47.226898 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.226869    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ae0e243-00fc-4b45-85e9-52d527e5fb25-kube-api-access-ps5b6" pod "4ae0e243-00fc-4b45-85e9-52d527e5fb25" (UID: "4ae0e243-00fc-4b45-85e9-52d527e5fb25"). InnerVolumeSpecName "kube-api-access-ps5b6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:47.228091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.228052    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-xks-ca" pod "4ae0e243-00fc-4b45-85e9-52d527e5fb25" (UID: "4ae0e243-00fc-4b45-85e9-52d527e5fb25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:47.325610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.325570    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps5b6\" (UniqueName: \"kubernetes.io/projected/4ae0e243-00fc-4b45-85e9-52d527e5fb25-kube-api-access-ps5b6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:47.325610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.325604    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ae0e243-00fc-4b45-85e9-52d527e5fb25-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:47.988386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.988356    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"
Oct 30 13:41:47.988554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.988388    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq" event={"ID":"4ae0e243-00fc-4b45-85e9-52d527e5fb25","Type":"ContainerDied","Data":"fadd0f88677c1e88d39de91a28c831333eaeca055ac675748aecbdd24151f80b"}
Oct 30 13:41:47.988554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:47.988443    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fadd0f88677c1e88d39de91a28c831333eaeca055ac675748aecbdd24151f80b"
Oct 30 13:41:51.079388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.079350    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"]
Oct 30 13:41:51.080261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.080236    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4ae0e243-00fc-4b45-85e9-52d527e5fb25" containerName="revoke-keys"
Oct 30 13:41:51.080373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.080263    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ae0e243-00fc-4b45-85e9-52d527e5fb25" containerName="revoke-keys"
Oct 30 13:41:51.080450 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.080435    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ae0e243-00fc-4b45-85e9-52d527e5fb25" containerName="revoke-keys"
Oct 30 13:41:51.084258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.084236    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.101418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.101387    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"]
Oct 30 13:41:51.109796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.109726    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cad2c6\""
Oct 30 13:41:51.109796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.109746    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"kube-root-ca.crt\""
Oct 30 13:41:51.109796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.109743    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-e2e--8d6d5ae6-dockercfg-wc9k6\""
Oct 30 13:41:51.110354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.110319    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"e2e-cmp-legacy-fddcb8-provider-cred\""
Oct 30 13:41:51.110354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.110323    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"openshift-service-ca.crt\""
Oct 30 13:41:51.110501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.110363    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"routing-overlay\""
Oct 30 13:41:51.261659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.259242    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nw4n\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-kube-api-access-4nw4n\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.261659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.259433    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.261659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.259507    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.261659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.259700    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.370213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.361601    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.370213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.361737    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nw4n\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-kube-api-access-4nw4n\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.370213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.361904    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.370213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.361987    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.370213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.364537    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.370213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.365298    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.371878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.370919    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.386048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.386021    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nw4n\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-kube-api-access-4nw4n\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.401865 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.401831    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:51.712927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.712898    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"]
Oct 30 13:41:51.714886 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:41:51.714838    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4def8600_3f7a_4c69_a6dc_9fefdc97cf3c.slice/crio-4364395b0755135dd0f3373589d18f1bc540641af29c1e8f38dad61a736eb0d0 WatchSource:0}: Error finding container 4364395b0755135dd0f3373589d18f1bc540641af29c1e8f38dad61a736eb0d0: Status 404 returned error can't find the container with id 4364395b0755135dd0f3373589d18f1bc540641af29c1e8f38dad61a736eb0d0
Oct 30 13:41:51.898471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:51.896745    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm" podUID="9d531323-5434-472e-a5c3-5a40238f0ee2" containerName="payload-processing" containerID="cri-o://d8ee8ea382fd038e0b0be59a6af545aaebe33fb38ace4c893d1e619eb0d08aa3" gracePeriod=25
Oct 30 13:41:52.033137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.033026    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x" event={"ID":"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c","Type":"ContainerStarted","Data":"fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e"}
Oct 30 13:41:52.033137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.033082    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x" event={"ID":"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c","Type":"ContainerStarted","Data":"4364395b0755135dd0f3373589d18f1bc540641af29c1e8f38dad61a736eb0d0"}
Oct 30 13:41:52.033137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.033106    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:41:52.034928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.034898    4291 generic.go:356] "Generic (PLEG): container finished" podID="9d531323-5434-472e-a5c3-5a40238f0ee2" containerID="d8ee8ea382fd038e0b0be59a6af545aaebe33fb38ace4c893d1e619eb0d08aa3" exitCode=0
Oct 30 13:41:52.035096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.034974    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm" event={"ID":"9d531323-5434-472e-a5c3-5a40238f0ee2","Type":"ContainerDied","Data":"d8ee8ea382fd038e0b0be59a6af545aaebe33fb38ace4c893d1e619eb0d08aa3"}
Oct 30 13:41:52.148739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.148716    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:52.210194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.210075    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x" podStartSLOduration=1.210048421 podStartE2EDuration="1.210048421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:52.080623162 +0000 UTC m=+2322.340861396" watchObservedRunningTime="2026-10-30 13:41:52.210048421 +0000 UTC m=+2322.470286655"
Oct 30 13:41:52.282109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.282023    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-provider-credentials\") pod \"9d531323-5434-472e-a5c3-5a40238f0ee2\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") "
Oct 30 13:41:52.282109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.282108    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-plugins-config-volume\") pod \"9d531323-5434-472e-a5c3-5a40238f0ee2\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") "
Oct 30 13:41:52.282312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.282270    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-kube-api-access-ng7sh\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-kube-api-access-ng7sh\") pod \"9d531323-5434-472e-a5c3-5a40238f0ee2\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") "
Oct 30 13:41:52.282348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.282333    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-routing-overlay-volume\") pod \"9d531323-5434-472e-a5c3-5a40238f0ee2\" (UID: \"9d531323-5434-472e-a5c3-5a40238f0ee2\") "
Oct 30 13:41:52.282669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.282536    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-plugins-config-volume" pod "9d531323-5434-472e-a5c3-5a40238f0ee2" (UID: "9d531323-5434-472e-a5c3-5a40238f0ee2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:52.282669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.282704    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-routing-overlay-volume" pod "9d531323-5434-472e-a5c3-5a40238f0ee2" (UID: "9d531323-5434-472e-a5c3-5a40238f0ee2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:52.283027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.283006    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:52.283077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.283033    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9d531323-5434-472e-a5c3-5a40238f0ee2-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:52.284206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.284180    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-provider-credentials" pod "9d531323-5434-472e-a5c3-5a40238f0ee2" (UID: "9d531323-5434-472e-a5c3-5a40238f0ee2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:52.284503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.284482    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-kube-api-access-ng7sh" pod "9d531323-5434-472e-a5c3-5a40238f0ee2" (UID: "9d531323-5434-472e-a5c3-5a40238f0ee2"). InnerVolumeSpecName "kube-api-access-ng7sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:52.384914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.384883    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng7sh\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-kube-api-access-ng7sh\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:52.384914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:52.384911    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9d531323-5434-472e-a5c3-5a40238f0ee2-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:53.043394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:53.043362    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"
Oct 30 13:41:53.043394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:53.043376    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm" event={"ID":"9d531323-5434-472e-a5c3-5a40238f0ee2","Type":"ContainerDied","Data":"7c62109c73b24765ca5d66471c05aeac59b8f0ca0d8abf987af3db2ff3f70497"}
Oct 30 13:41:53.043716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:53.043433    4291 scope.go:122] "RemoveContainer" containerID="d8ee8ea382fd038e0b0be59a6af545aaebe33fb38ace4c893d1e619eb0d08aa3"
Oct 30 13:41:53.095040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:53.095004    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"]
Oct 30 13:41:53.106712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:53.106673    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-6c8fhm2zm"]
Oct 30 13:41:53.879956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:53.879921    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d531323-5434-472e-a5c3-5a40238f0ee2" path="/var/lib/kubelet/pods/9d531323-5434-472e-a5c3-5a40238f0ee2/volumes"
Oct 30 13:41:55.401661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.401477    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"]
Oct 30 13:41:55.405778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.402448    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9d531323-5434-472e-a5c3-5a40238f0ee2" containerName="payload-processing"
Oct 30 13:41:55.405778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.402472    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d531323-5434-472e-a5c3-5a40238f0ee2" containerName="payload-processing"
Oct 30 13:41:55.405778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.402628    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d531323-5434-472e-a5c3-5a40238f0ee2" containerName="payload-processing"
Oct 30 13:41:55.429853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.429818    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"]
Oct 30 13:41:55.430242 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.430163    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.457682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.457629    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:41:55.458003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.457987    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:41:55.458195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.458181    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:41:55.458364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.458351    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:41:55.543239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.543194    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.543427 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.543323    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.543427 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.543379    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mjnf\" (UniqueName: \"kubernetes.io/projected/9003bb56-77a1-40d8-80ff-5727da7fc74e-kube-api-access-2mjnf\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.644938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.644899    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.645134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.644953    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mjnf\" (UniqueName: \"kubernetes.io/projected/9003bb56-77a1-40d8-80ff-5727da7fc74e-kube-api-access-2mjnf\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.645134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.645119    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.645817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.645789    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.647542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.647513    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.684352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.684272    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mjnf\" (UniqueName: \"kubernetes.io/projected/9003bb56-77a1-40d8-80ff-5727da7fc74e-kube-api-access-2mjnf\") pod \"maas-api-revoke-sub-cb684c97-r9hmj\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:55.811919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:55.811875    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:56.118403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:56.118374    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"]
Oct 30 13:41:56.119461 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:41:56.119428    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9003bb56_77a1_40d8_80ff_5727da7fc74e.slice/crio-da6e58baac8a8633486237e89f23b9066d6793a93c7378da392afbba5b614ab0 WatchSource:0}: Error finding container da6e58baac8a8633486237e89f23b9066d6793a93c7378da392afbba5b614ab0: Status 404 returned error can't find the container with id da6e58baac8a8633486237e89f23b9066d6793a93c7378da392afbba5b614ab0
Oct 30 13:41:57.074549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:57.074505    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj" event={"ID":"9003bb56-77a1-40d8-80ff-5727da7fc74e","Type":"ContainerStarted","Data":"624bc313313d9a36d726e8d829d1045c9f76bba1d23d7579d98763904b7e73cf"}
Oct 30 13:41:57.075008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:57.074584    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj" event={"ID":"9003bb56-77a1-40d8-80ff-5727da7fc74e","Type":"ContainerStarted","Data":"da6e58baac8a8633486237e89f23b9066d6793a93c7378da392afbba5b614ab0"}
Oct 30 13:41:57.115523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:57.115458    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj" podStartSLOduration=2.115439863 podStartE2EDuration="2.115439863s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:41:57.111379846 +0000 UTC m=+2327.371618071" watchObservedRunningTime="2026-10-30 13:41:57.115439863 +0000 UTC m=+2327.375678114"
Oct 30 13:41:58.083213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:58.083177    4291 generic.go:356] "Generic (PLEG): container finished" podID="9003bb56-77a1-40d8-80ff-5727da7fc74e" containerID="624bc313313d9a36d726e8d829d1045c9f76bba1d23d7579d98763904b7e73cf" exitCode=0
Oct 30 13:41:58.083707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:58.083258    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj" event={"ID":"9003bb56-77a1-40d8-80ff-5727da7fc74e","Type":"ContainerDied","Data":"624bc313313d9a36d726e8d829d1045c9f76bba1d23d7579d98763904b7e73cf"}
Oct 30 13:41:59.260384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.260349    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:41:59.405016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.404972    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-openshift-ca\") pod \"9003bb56-77a1-40d8-80ff-5727da7fc74e\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") "
Oct 30 13:41:59.405217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.405140    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-xks-ca\") pod \"9003bb56-77a1-40d8-80ff-5727da7fc74e\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") "
Oct 30 13:41:59.405217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.405183    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9003bb56-77a1-40d8-80ff-5727da7fc74e-kube-api-access-2mjnf\" (UniqueName: \"kubernetes.io/projected/9003bb56-77a1-40d8-80ff-5727da7fc74e-kube-api-access-2mjnf\") pod \"9003bb56-77a1-40d8-80ff-5727da7fc74e\" (UID: \"9003bb56-77a1-40d8-80ff-5727da7fc74e\") "
Oct 30 13:41:59.405456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.405430    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-openshift-ca" pod "9003bb56-77a1-40d8-80ff-5727da7fc74e" (UID: "9003bb56-77a1-40d8-80ff-5727da7fc74e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:41:59.405565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.405550    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:59.407424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.407390    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-xks-ca" pod "9003bb56-77a1-40d8-80ff-5727da7fc74e" (UID: "9003bb56-77a1-40d8-80ff-5727da7fc74e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:41:59.407424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.407400    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9003bb56-77a1-40d8-80ff-5727da7fc74e-kube-api-access-2mjnf" pod "9003bb56-77a1-40d8-80ff-5727da7fc74e" (UID: "9003bb56-77a1-40d8-80ff-5727da7fc74e"). InnerVolumeSpecName "kube-api-access-2mjnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:41:59.507268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.507231    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mjnf\" (UniqueName: \"kubernetes.io/projected/9003bb56-77a1-40d8-80ff-5727da7fc74e-kube-api-access-2mjnf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:41:59.507268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:41:59.507266    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9003bb56-77a1-40d8-80ff-5727da7fc74e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:00.099206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:00.099172    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"
Oct 30 13:42:00.099206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:00.099183    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj" event={"ID":"9003bb56-77a1-40d8-80ff-5727da7fc74e","Type":"ContainerDied","Data":"da6e58baac8a8633486237e89f23b9066d6793a93c7378da392afbba5b614ab0"}
Oct 30 13:42:00.099427 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:00.099228    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da6e58baac8a8633486237e89f23b9066d6793a93c7378da392afbba5b614ab0"
Oct 30 13:42:03.047580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:03.047547    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:42:07.568845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.568505    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"]
Oct 30 13:42:07.571433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.571402    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9003bb56-77a1-40d8-80ff-5727da7fc74e" containerName="revoke-keys"
Oct 30 13:42:07.571668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.571654    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9003bb56-77a1-40d8-80ff-5727da7fc74e" containerName="revoke-keys"
Oct 30 13:42:07.571989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.571973    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9003bb56-77a1-40d8-80ff-5727da7fc74e" containerName="revoke-keys"
Oct 30 13:42:07.577020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.576994    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.595919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.592521    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"]
Oct 30 13:42:07.602458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.602318    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:42:07.602458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.602449    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:42:07.602770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.602520    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:07.604878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.604853    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:42:07.677341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.677292    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.677556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.677368    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.677556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.677459    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rml7\" (UniqueName: \"kubernetes.io/projected/042745cf-d3cc-4c6a-a34b-126efbf9b385-kube-api-access-6rml7\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.778529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.778488    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.778856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.778582    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.778856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.778681    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rml7\" (UniqueName: \"kubernetes.io/projected/042745cf-d3cc-4c6a-a34b-126efbf9b385-kube-api-access-6rml7\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.779316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.779294    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.781093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.781074    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.797987 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.797948    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rml7\" (UniqueName: \"kubernetes.io/projected/042745cf-d3cc-4c6a-a34b-126efbf9b385-kube-api-access-6rml7\") pod \"maas-api-revoke-sub-efa0936c-rslsw\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:07.915778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:07.915741    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:08.210498 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:42:08.210417    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:42:08.219095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:08.219070    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"]
Oct 30 13:42:08.222123 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:42:08.221985    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod042745cf_d3cc_4c6a_a34b_126efbf9b385.slice/crio-02c27c6e275faa2ac2e091c79bc25106f5bee6115288a73026cc6c32ca75f118 WatchSource:0}: Error finding container 02c27c6e275faa2ac2e091c79bc25106f5bee6115288a73026cc6c32ca75f118: Status 404 returned error can't find the container with id 02c27c6e275faa2ac2e091c79bc25106f5bee6115288a73026cc6c32ca75f118
Oct 30 13:42:09.176729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:09.176695    4291 generic.go:356] "Generic (PLEG): container finished" podID="042745cf-d3cc-4c6a-a34b-126efbf9b385" containerID="5c48243fcfcc1789f370f48bf15f6f9c4471c9104953c7dd16a0a4dcc28a5127" exitCode=0
Oct 30 13:42:09.177123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:09.176740    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw" event={"ID":"042745cf-d3cc-4c6a-a34b-126efbf9b385","Type":"ContainerDied","Data":"5c48243fcfcc1789f370f48bf15f6f9c4471c9104953c7dd16a0a4dcc28a5127"}
Oct 30 13:42:09.177123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:09.176779    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw" event={"ID":"042745cf-d3cc-4c6a-a34b-126efbf9b385","Type":"ContainerStarted","Data":"02c27c6e275faa2ac2e091c79bc25106f5bee6115288a73026cc6c32ca75f118"}
Oct 30 13:42:10.377368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.377341    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:10.433224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.433178    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-openshift-ca\") pod \"042745cf-d3cc-4c6a-a34b-126efbf9b385\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") "
Oct 30 13:42:10.433410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.433291    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/042745cf-d3cc-4c6a-a34b-126efbf9b385-kube-api-access-6rml7\" (UniqueName: \"kubernetes.io/projected/042745cf-d3cc-4c6a-a34b-126efbf9b385-kube-api-access-6rml7\") pod \"042745cf-d3cc-4c6a-a34b-126efbf9b385\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") "
Oct 30 13:42:10.433512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.433446    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-xks-ca\") pod \"042745cf-d3cc-4c6a-a34b-126efbf9b385\" (UID: \"042745cf-d3cc-4c6a-a34b-126efbf9b385\") "
Oct 30 13:42:10.433620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.433589    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-openshift-ca" pod "042745cf-d3cc-4c6a-a34b-126efbf9b385" (UID: "042745cf-d3cc-4c6a-a34b-126efbf9b385"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:10.434054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.434031    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:10.436148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.436087    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-xks-ca" pod "042745cf-d3cc-4c6a-a34b-126efbf9b385" (UID: "042745cf-d3cc-4c6a-a34b-126efbf9b385"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:10.436148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.436105    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/042745cf-d3cc-4c6a-a34b-126efbf9b385-kube-api-access-6rml7" pod "042745cf-d3cc-4c6a-a34b-126efbf9b385" (UID: "042745cf-d3cc-4c6a-a34b-126efbf9b385"). InnerVolumeSpecName "kube-api-access-6rml7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:10.535459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.535419    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rml7\" (UniqueName: \"kubernetes.io/projected/042745cf-d3cc-4c6a-a34b-126efbf9b385-kube-api-access-6rml7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:10.535459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:10.535454    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/042745cf-d3cc-4c6a-a34b-126efbf9b385-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:11.192215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:11.192186    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"
Oct 30 13:42:11.192377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:11.192223    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw" event={"ID":"042745cf-d3cc-4c6a-a34b-126efbf9b385","Type":"ContainerDied","Data":"02c27c6e275faa2ac2e091c79bc25106f5bee6115288a73026cc6c32ca75f118"}
Oct 30 13:42:11.192377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:11.192272    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c27c6e275faa2ac2e091c79bc25106f5bee6115288a73026cc6c32ca75f118"
Oct 30 13:42:12.562500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:12.562468    4291 scope.go:122] "RemoveContainer" containerID="0a6ed0ec41bb0286034993dfc5aa022bcb0b1a0f3014fbb7910f145882a94542"
Oct 30 13:42:12.572323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:12.572303    4291 scope.go:122] "RemoveContainer" containerID="5a1d814f023f60b107ce07d08505203fcd579f148b01442388bd50e6e0c5ae2a"
Oct 30 13:42:12.581211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:12.581183    4291 scope.go:122] "RemoveContainer" containerID="39a945d8adbbbd4f461b64b4bedef5303f0d500c38618dc7342cb0da34911994"
Oct 30 13:42:12.590665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:12.590631    4291 scope.go:122] "RemoveContainer" containerID="c0cc03468771aff165bf897edb9546fe047da49a3e7e42097c78c835a14abda9"
Oct 30 13:42:31.919550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:31.919515    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:42:38.213754 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:42:38.213712    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:42:49.381920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.381885    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"]
Oct 30 13:42:49.382532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.382516    4291 cpu_manager.go:414] "containerMap: removing container" podUID="042745cf-d3cc-4c6a-a34b-126efbf9b385" containerName="revoke-keys"
Oct 30 13:42:49.382580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.382532    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="042745cf-d3cc-4c6a-a34b-126efbf9b385" containerName="revoke-keys"
Oct 30 13:42:49.382669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.382660    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="042745cf-d3cc-4c6a-a34b-126efbf9b385" containerName="revoke-keys"
Oct 30 13:42:49.386487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.386468    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.405443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.405414    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"]
Oct 30 13:42:49.412433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.412405    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:42:49.412659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.412479    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:42:49.412659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.412508    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:42:49.412796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.412677    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:42:49.542060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.542024    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.542280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.542183    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.542280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.542238    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n25bc\" (UniqueName: \"kubernetes.io/projected/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-kube-api-access-n25bc\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.643476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.643382    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.643476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.643435    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n25bc\" (UniqueName: \"kubernetes.io/projected/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-kube-api-access-n25bc\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.643793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.643566    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.644196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.644171    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.645947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.645922    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.662993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.662966    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n25bc\" (UniqueName: \"kubernetes.io/projected/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-kube-api-access-n25bc\") pod \"maas-api-revoke-sub-5ed1e354-bj2xg\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.700088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.700053    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:49.928794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:49.928708    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"]
Oct 30 13:42:50.021766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:50.021733    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"]
Oct 30 13:42:50.024370 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:42:50.024342    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a1b052e_5ea8_4fd3_b039_d8e086ef268c.slice/crio-4bfa16bdc357e3dde9622317996f4c882b473f106591e45296359f3f803b65c3 WatchSource:0}: Error finding container 4bfa16bdc357e3dde9622317996f4c882b473f106591e45296359f3f803b65c3: Status 404 returned error can't find the container with id 4bfa16bdc357e3dde9622317996f4c882b473f106591e45296359f3f803b65c3
Oct 30 13:42:50.486398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:50.486361    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg" event={"ID":"1a1b052e-5ea8-4fd3-b039-d8e086ef268c","Type":"ContainerStarted","Data":"3fb79cf00dc57ebb2af59e818b360310b16b95289d8cf9bac6d33e6395347db0"}
Oct 30 13:42:50.486822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:50.486423    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg" event={"ID":"1a1b052e-5ea8-4fd3-b039-d8e086ef268c","Type":"ContainerStarted","Data":"4bfa16bdc357e3dde9622317996f4c882b473f106591e45296359f3f803b65c3"}
Oct 30 13:42:50.531553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:50.531450    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg" podStartSLOduration=1.531434696 podStartE2EDuration="1.531434696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:42:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:42:50.530291228 +0000 UTC m=+2380.790529462" watchObservedRunningTime="2026-10-30 13:42:50.531434696 +0000 UTC m=+2380.791672927"
Oct 30 13:42:51.494689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:51.494651    4291 generic.go:356] "Generic (PLEG): container finished" podID="1a1b052e-5ea8-4fd3-b039-d8e086ef268c" containerID="3fb79cf00dc57ebb2af59e818b360310b16b95289d8cf9bac6d33e6395347db0" exitCode=0
Oct 30 13:42:51.495111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:51.494732    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg" event={"ID":"1a1b052e-5ea8-4fd3-b039-d8e086ef268c","Type":"ContainerDied","Data":"3fb79cf00dc57ebb2af59e818b360310b16b95289d8cf9bac6d33e6395347db0"}
Oct 30 13:42:52.663992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.663965    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:52.694984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.694949    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-xks-ca\") pod \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") "
Oct 30 13:42:52.695193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.695169    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-openshift-ca\") pod \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") "
Oct 30 13:42:52.695270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.695246    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-kube-api-access-n25bc\" (UniqueName: \"kubernetes.io/projected/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-kube-api-access-n25bc\") pod \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\" (UID: \"1a1b052e-5ea8-4fd3-b039-d8e086ef268c\") "
Oct 30 13:42:52.695491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.695467    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-openshift-ca" pod "1a1b052e-5ea8-4fd3-b039-d8e086ef268c" (UID: "1a1b052e-5ea8-4fd3-b039-d8e086ef268c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:52.695853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.695835    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.697092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.697071    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-xks-ca" pod "1a1b052e-5ea8-4fd3-b039-d8e086ef268c" (UID: "1a1b052e-5ea8-4fd3-b039-d8e086ef268c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:42:52.697230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.697209    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-kube-api-access-n25bc" pod "1a1b052e-5ea8-4fd3-b039-d8e086ef268c" (UID: "1a1b052e-5ea8-4fd3-b039-d8e086ef268c"). InnerVolumeSpecName "kube-api-access-n25bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:52.797131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.796995    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n25bc\" (UniqueName: \"kubernetes.io/projected/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-kube-api-access-n25bc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:52.797131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:52.797045    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a1b052e-5ea8-4fd3-b039-d8e086ef268c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:53.509809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:53.509775    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"
Oct 30 13:42:53.510000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:53.509805    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg" event={"ID":"1a1b052e-5ea8-4fd3-b039-d8e086ef268c","Type":"ContainerDied","Data":"4bfa16bdc357e3dde9622317996f4c882b473f106591e45296359f3f803b65c3"}
Oct 30 13:42:53.510000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:53.509860    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bfa16bdc357e3dde9622317996f4c882b473f106591e45296359f3f803b65c3"
Oct 30 13:42:54.929664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:54.929612    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2" podUID="54221ff6-6615-4dfd-8590-17b47a9121c9" containerName="payload-processing" containerID="cri-o://d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f" gracePeriod=25
Oct 30 13:42:55.208896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.208865    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:42:55.332027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.331984    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-plugins-config-volume\") pod \"54221ff6-6615-4dfd-8590-17b47a9121c9\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") "
Oct 30 13:42:55.332239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.332039    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-routing-overlay-volume\") pod \"54221ff6-6615-4dfd-8590-17b47a9121c9\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") "
Oct 30 13:42:55.332239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.332157    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-kube-api-access-vz8fk\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-kube-api-access-vz8fk\") pod \"54221ff6-6615-4dfd-8590-17b47a9121c9\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") "
Oct 30 13:42:55.332239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.332219    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-provider-credentials\") pod \"54221ff6-6615-4dfd-8590-17b47a9121c9\" (UID: \"54221ff6-6615-4dfd-8590-17b47a9121c9\") "
Oct 30 13:42:55.332487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.332455    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-routing-overlay-volume" pod "54221ff6-6615-4dfd-8590-17b47a9121c9" (UID: "54221ff6-6615-4dfd-8590-17b47a9121c9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:55.332559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.332465    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-plugins-config-volume" pod "54221ff6-6615-4dfd-8590-17b47a9121c9" (UID: "54221ff6-6615-4dfd-8590-17b47a9121c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:42:55.332856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.332835    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:55.332940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.332860    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/54221ff6-6615-4dfd-8590-17b47a9121c9-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:55.334490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.334467    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-provider-credentials" pod "54221ff6-6615-4dfd-8590-17b47a9121c9" (UID: "54221ff6-6615-4dfd-8590-17b47a9121c9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:55.334585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.334541    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-kube-api-access-vz8fk" pod "54221ff6-6615-4dfd-8590-17b47a9121c9" (UID: "54221ff6-6615-4dfd-8590-17b47a9121c9"). InnerVolumeSpecName "kube-api-access-vz8fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:42:55.434303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.434255    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:55.434303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.434293    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vz8fk\" (UniqueName: \"kubernetes.io/projected/54221ff6-6615-4dfd-8590-17b47a9121c9-kube-api-access-vz8fk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:42:55.524531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.524433    4291 generic.go:356] "Generic (PLEG): container finished" podID="54221ff6-6615-4dfd-8590-17b47a9121c9" containerID="d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f" exitCode=0
Oct 30 13:42:55.524531 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.524501    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"
Oct 30 13:42:55.524842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.524517    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2" event={"ID":"54221ff6-6615-4dfd-8590-17b47a9121c9","Type":"ContainerDied","Data":"d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f"}
Oct 30 13:42:55.524842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.524620    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2" event={"ID":"54221ff6-6615-4dfd-8590-17b47a9121c9","Type":"ContainerDied","Data":"f2d098933de5e19c624c471c5561ee50a94230336699bda89a999b606e13102c"}
Oct 30 13:42:55.524842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.524684    4291 scope.go:122] "RemoveContainer" containerID="d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f"
Oct 30 13:42:55.535791 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.535760    4291 scope.go:122] "RemoveContainer" containerID="d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f"
Oct 30 13:42:55.536096 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:42:55.536072    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f\": container with ID starting with d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f not found: ID does not exist" containerID="d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f"
Oct 30 13:42:55.536176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.536104    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f"} err="failed to get container status \"d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f\": rpc error: code = NotFound desc = could not find container \"d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f\": container with ID starting with d7e9ef7d53727d7f86ec0e35e994c25d2794199c3b541dcfc11687b989f1189f not found: ID does not exist"
Oct 30 13:42:55.576870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.576839    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"]
Oct 30 13:42:55.580134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.580111    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-6cd7rzvv2"]
Oct 30 13:42:55.880567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:42:55.880528    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54221ff6-6615-4dfd-8590-17b47a9121c9" path="/var/lib/kubelet/pods/54221ff6-6615-4dfd-8590-17b47a9121c9/volumes"
Oct 30 13:43:08.211705 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:43:08.211669    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:43:12.651025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:12.650994    4291 scope.go:122] "RemoveContainer" containerID="1187ab5129d8be2dde694ca97a620e4e275415f7033a80acf9f9f1e9aa1535e8"
Oct 30 13:43:12.660762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:12.660737    4291 scope.go:122] "RemoveContainer" containerID="cbc6caeeb985d9df7aa6c3b9eeefd467b2feb8783b1b52cf1cd5215291fcc42e"
Oct 30 13:43:22.510869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.510838    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"]
Oct 30 13:43:22.511471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.511455    4291 cpu_manager.go:414] "containerMap: removing container" podUID="54221ff6-6615-4dfd-8590-17b47a9121c9" containerName="payload-processing"
Oct 30 13:43:22.511525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.511471    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54221ff6-6615-4dfd-8590-17b47a9121c9" containerName="payload-processing"
Oct 30 13:43:22.511525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.511495    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1a1b052e-5ea8-4fd3-b039-d8e086ef268c" containerName="revoke-keys"
Oct 30 13:43:22.511525 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.511500    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a1b052e-5ea8-4fd3-b039-d8e086ef268c" containerName="revoke-keys"
Oct 30 13:43:22.511622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.511583    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="54221ff6-6615-4dfd-8590-17b47a9121c9" containerName="payload-processing"
Oct 30 13:43:22.511622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.511595    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a1b052e-5ea8-4fd3-b039-d8e086ef268c" containerName="revoke-keys"
Oct 30 13:43:22.517063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.517044    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.531596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.531571    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"]
Oct 30 13:43:22.539341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.539084    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:43:22.539341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.539173    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:43:22.539341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.539081    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:43:22.539341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.539292    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:43:22.572560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.572505    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/98ae2010-d727-4e48-a71b-0082a5005a62-kube-api-access-g8q2g\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.572785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.572760    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.572905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.572882    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.674580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.674538    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.674843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.674608    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.674843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.674751    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/98ae2010-d727-4e48-a71b-0082a5005a62-kube-api-access-g8q2g\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.675372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.675338    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.677225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.677201    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.694043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.694008    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/98ae2010-d727-4e48-a71b-0082a5005a62-kube-api-access-g8q2g\") pod \"maas-api-revoke-sub-f6493b38-97zp4\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:22.833046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:22.832950    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:23.141764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:23.141738    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"]
Oct 30 13:43:23.142882 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:43:23.142849    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98ae2010_d727_4e48_a71b_0082a5005a62.slice/crio-90b7e8badb8adcc0515a06d8ac18a9b9d729711e450a6a4c2c5a5e68f3d27e86 WatchSource:0}: Error finding container 90b7e8badb8adcc0515a06d8ac18a9b9d729711e450a6a4c2c5a5e68f3d27e86: Status 404 returned error can't find the container with id 90b7e8badb8adcc0515a06d8ac18a9b9d729711e450a6a4c2c5a5e68f3d27e86
Oct 30 13:43:23.751611 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:23.751570    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4" event={"ID":"98ae2010-d727-4e48-a71b-0082a5005a62","Type":"ContainerStarted","Data":"bd655b0dfc0a59413e42fc4c470f7651184f59f0485f87efad529defd1b61389"}
Oct 30 13:43:23.752100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:23.751667    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4" event={"ID":"98ae2010-d727-4e48-a71b-0082a5005a62","Type":"ContainerStarted","Data":"90b7e8badb8adcc0515a06d8ac18a9b9d729711e450a6a4c2c5a5e68f3d27e86"}
Oct 30 13:43:23.793552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:23.793454    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4" podStartSLOduration=1.793427299 podStartE2EDuration="1.793427299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:23.790468643 +0000 UTC m=+2414.050706876" watchObservedRunningTime="2026-10-30 13:43:23.793427299 +0000 UTC m=+2414.053665535"
Oct 30 13:43:24.761177 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:24.761146    4291 generic.go:356] "Generic (PLEG): container finished" podID="98ae2010-d727-4e48-a71b-0082a5005a62" containerID="bd655b0dfc0a59413e42fc4c470f7651184f59f0485f87efad529defd1b61389" exitCode=0
Oct 30 13:43:24.761601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:24.761282    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4" event={"ID":"98ae2010-d727-4e48-a71b-0082a5005a62","Type":"ContainerDied","Data":"bd655b0dfc0a59413e42fc4c470f7651184f59f0485f87efad529defd1b61389"}
Oct 30 13:43:25.941400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:25.941372    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:26.021053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.020954    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-openshift-ca\") pod \"98ae2010-d727-4e48-a71b-0082a5005a62\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") "
Oct 30 13:43:26.021225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.021134    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-xks-ca\") pod \"98ae2010-d727-4e48-a71b-0082a5005a62\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") "
Oct 30 13:43:26.021296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.021231    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98ae2010-d727-4e48-a71b-0082a5005a62-kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/98ae2010-d727-4e48-a71b-0082a5005a62-kube-api-access-g8q2g\") pod \"98ae2010-d727-4e48-a71b-0082a5005a62\" (UID: \"98ae2010-d727-4e48-a71b-0082a5005a62\") "
Oct 30 13:43:26.021364 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.021334    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-openshift-ca" pod "98ae2010-d727-4e48-a71b-0082a5005a62" (UID: "98ae2010-d727-4e48-a71b-0082a5005a62"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:26.022493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.022438    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:26.023647 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.023602    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-xks-ca" pod "98ae2010-d727-4e48-a71b-0082a5005a62" (UID: "98ae2010-d727-4e48-a71b-0082a5005a62"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:26.023775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.023651    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98ae2010-d727-4e48-a71b-0082a5005a62-kube-api-access-g8q2g" pod "98ae2010-d727-4e48-a71b-0082a5005a62" (UID: "98ae2010-d727-4e48-a71b-0082a5005a62"). InnerVolumeSpecName "kube-api-access-g8q2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:26.124415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.124374    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98ae2010-d727-4e48-a71b-0082a5005a62-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:26.124415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.124415    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8q2g\" (UniqueName: \"kubernetes.io/projected/98ae2010-d727-4e48-a71b-0082a5005a62-kube-api-access-g8q2g\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:26.776715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.776681    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"
Oct 30 13:43:26.776715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.776697    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4" event={"ID":"98ae2010-d727-4e48-a71b-0082a5005a62","Type":"ContainerDied","Data":"90b7e8badb8adcc0515a06d8ac18a9b9d729711e450a6a4c2c5a5e68f3d27e86"}
Oct 30 13:43:26.776955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:26.776746    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90b7e8badb8adcc0515a06d8ac18a9b9d729711e450a6a4c2c5a5e68f3d27e86"
Oct 30 13:43:34.018651 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.018600    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"]
Oct 30 13:43:34.019216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.019202    4291 cpu_manager.go:414] "containerMap: removing container" podUID="98ae2010-d727-4e48-a71b-0082a5005a62" containerName="revoke-keys"
Oct 30 13:43:34.019216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.019217    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98ae2010-d727-4e48-a71b-0082a5005a62" containerName="revoke-keys"
Oct 30 13:43:34.019350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.019341    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="98ae2010-d727-4e48-a71b-0082a5005a62" containerName="revoke-keys"
Oct 30 13:43:34.022962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.022939    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.037489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.037459    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kxfmq\" (UniqueName: \"kubernetes.io/projected/f94f1fec-4ad4-4699-84a4-2bb25df26896-kube-api-access-kxfmq\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.037656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.037498    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.037656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.037529    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.039168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.039146    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"]
Oct 30 13:43:34.047068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.046947    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:43:34.047068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.046998    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:43:34.047068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.046999    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:43:34.051010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.050991    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:43:34.138909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.138870    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kxfmq\" (UniqueName: \"kubernetes.io/projected/f94f1fec-4ad4-4699-84a4-2bb25df26896-kube-api-access-kxfmq\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.139111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.138916    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.139111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.138966    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.139713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.139684    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.141361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.141343    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.159554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.159529    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kxfmq\" (UniqueName: \"kubernetes.io/projected/f94f1fec-4ad4-4699-84a4-2bb25df26896-kube-api-access-kxfmq\") pod \"maas-api-revoke-sub-fd7ac126-kzgrw\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.337206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.337121    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:34.351824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.350814    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:43:34.653350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.653318    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"]
Oct 30 13:43:34.654595 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:43:34.654565    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf94f1fec_4ad4_4699_84a4_2bb25df26896.slice/crio-785fc0169e1141537a1155f4f8677f00f3c780cebfa6454f8d25976d29abb33e WatchSource:0}: Error finding container 785fc0169e1141537a1155f4f8677f00f3c780cebfa6454f8d25976d29abb33e: Status 404 returned error can't find the container with id 785fc0169e1141537a1155f4f8677f00f3c780cebfa6454f8d25976d29abb33e
Oct 30 13:43:34.868200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:34.868161    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw" event={"ID":"f94f1fec-4ad4-4699-84a4-2bb25df26896","Type":"ContainerStarted","Data":"785fc0169e1141537a1155f4f8677f00f3c780cebfa6454f8d25976d29abb33e"}
Oct 30 13:43:35.879117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:35.879084    4291 generic.go:356] "Generic (PLEG): container finished" podID="f94f1fec-4ad4-4699-84a4-2bb25df26896" containerID="9e3d40c8b99aa8ab16b06a360e794732568492b0fa1d781857ef054205f37c99" exitCode=0
Oct 30 13:43:35.881444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:35.881416    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw" event={"ID":"f94f1fec-4ad4-4699-84a4-2bb25df26896","Type":"ContainerDied","Data":"9e3d40c8b99aa8ab16b06a360e794732568492b0fa1d781857ef054205f37c99"}
Oct 30 13:43:37.111074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.111041    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:37.185697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.185552    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f94f1fec-4ad4-4699-84a4-2bb25df26896-kube-api-access-kxfmq\" (UniqueName: \"kubernetes.io/projected/f94f1fec-4ad4-4699-84a4-2bb25df26896-kube-api-access-kxfmq\") pod \"f94f1fec-4ad4-4699-84a4-2bb25df26896\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") "
Oct 30 13:43:37.186460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.186434    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-xks-ca\") pod \"f94f1fec-4ad4-4699-84a4-2bb25df26896\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") "
Oct 30 13:43:37.186805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.186784    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-openshift-ca\") pod \"f94f1fec-4ad4-4699-84a4-2bb25df26896\" (UID: \"f94f1fec-4ad4-4699-84a4-2bb25df26896\") "
Oct 30 13:43:37.188088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.188063    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-openshift-ca" pod "f94f1fec-4ad4-4699-84a4-2bb25df26896" (UID: "f94f1fec-4ad4-4699-84a4-2bb25df26896"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:37.190542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.190497    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-xks-ca" pod "f94f1fec-4ad4-4699-84a4-2bb25df26896" (UID: "f94f1fec-4ad4-4699-84a4-2bb25df26896"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:37.191131 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.191091    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f94f1fec-4ad4-4699-84a4-2bb25df26896-kube-api-access-kxfmq" pod "f94f1fec-4ad4-4699-84a4-2bb25df26896" (UID: "f94f1fec-4ad4-4699-84a4-2bb25df26896"). InnerVolumeSpecName "kube-api-access-kxfmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:37.289494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.289444    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kxfmq\" (UniqueName: \"kubernetes.io/projected/f94f1fec-4ad4-4699-84a4-2bb25df26896-kube-api-access-kxfmq\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:37.289494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.289495    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:37.289749 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.289513    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f94f1fec-4ad4-4699-84a4-2bb25df26896-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:37.903664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.901328    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw" event={"ID":"f94f1fec-4ad4-4699-84a4-2bb25df26896","Type":"ContainerDied","Data":"785fc0169e1141537a1155f4f8677f00f3c780cebfa6454f8d25976d29abb33e"}
Oct 30 13:43:37.903664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.901382    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="785fc0169e1141537a1155f4f8677f00f3c780cebfa6454f8d25976d29abb33e"
Oct 30 13:43:37.903664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:37.901443    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"
Oct 30 13:43:38.220795 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:43:38.220685    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:43:44.601319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.601283    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"]
Oct 30 13:43:44.601957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.601940    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f94f1fec-4ad4-4699-84a4-2bb25df26896" containerName="revoke-keys"
Oct 30 13:43:44.602011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.601958    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f94f1fec-4ad4-4699-84a4-2bb25df26896" containerName="revoke-keys"
Oct 30 13:43:44.602083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.602073    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f94f1fec-4ad4-4699-84a4-2bb25df26896" containerName="revoke-keys"
Oct 30 13:43:44.605744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.605726    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.623128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.623099    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"]
Oct 30 13:43:44.630229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.630147    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:43:44.630229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.630177    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:43:44.630441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.630256    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:43:44.633034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.632998    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:43:44.693080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.693042    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9v4b\" (UniqueName: \"kubernetes.io/projected/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-kube-api-access-p9v4b\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.693280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.693195    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.693280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.693229    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.794447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.794404    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9v4b\" (UniqueName: \"kubernetes.io/projected/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-kube-api-access-p9v4b\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.794682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.794506    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.794682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.794531    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.795309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.795290    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.797004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.796979    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.813897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.813860    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9v4b\" (UniqueName: \"kubernetes.io/projected/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-kube-api-access-p9v4b\") pod \"maas-api-revoke-sub-bfb52394-cjt44\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.921502 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.921406    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:44.973330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.973290    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"]
Oct 30 13:43:44.977916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:44.977886    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.008118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.008080    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"]
Oct 30 13:43:45.099810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.099757    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm85s\" (UniqueName: \"kubernetes.io/projected/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-kube-api-access-lm85s\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.100035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.099908    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.100127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.100094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.201793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.201700    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm85s\" (UniqueName: \"kubernetes.io/projected/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-kube-api-access-lm85s\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.201957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.201868    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.202032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.202013    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.202735 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.202710    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.204435 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.204403    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.221894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.221862    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm85s\" (UniqueName: \"kubernetes.io/projected/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-kube-api-access-lm85s\") pod \"maas-api-revoke-sub-e4d88c78-7s44v\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.234167 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:43:45.234137    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0e25fa_ae47_4b5e_8df3_78e61c9d2e7e.slice/crio-ac50c6ab5d9744998d16bb4cdd7da4cc2f17f621999f30e8312fbbee28d091e8 WatchSource:0}: Error finding container ac50c6ab5d9744998d16bb4cdd7da4cc2f17f621999f30e8312fbbee28d091e8: Status 404 returned error can't find the container with id ac50c6ab5d9744998d16bb4cdd7da4cc2f17f621999f30e8312fbbee28d091e8
Oct 30 13:43:45.234295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.234263    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"]
Oct 30 13:43:45.298672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.298604    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:45.610252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.610219    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"]
Oct 30 13:43:45.611175 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:43:45.611147    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6d1b0f_2965_41ec_a32e_4a688d8542ef.slice/crio-7792bc1d6f0e698706fa92ea5e7de2a5b314382435faaca44032e12e0d28f85a WatchSource:0}: Error finding container 7792bc1d6f0e698706fa92ea5e7de2a5b314382435faaca44032e12e0d28f85a: Status 404 returned error can't find the container with id 7792bc1d6f0e698706fa92ea5e7de2a5b314382435faaca44032e12e0d28f85a
Oct 30 13:43:45.958112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.958071    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v" event={"ID":"8a6d1b0f-2965-41ec-a32e-4a688d8542ef","Type":"ContainerStarted","Data":"7792bc1d6f0e698706fa92ea5e7de2a5b314382435faaca44032e12e0d28f85a"}
Oct 30 13:43:45.959787 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.959757    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44" event={"ID":"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e","Type":"ContainerStarted","Data":"667874c1904adfba568a707c57add78f323de20bc98518da4cfa92cd117149e1"}
Oct 30 13:43:45.959921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:45.959819    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44" event={"ID":"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e","Type":"ContainerStarted","Data":"ac50c6ab5d9744998d16bb4cdd7da4cc2f17f621999f30e8312fbbee28d091e8"}
Oct 30 13:43:46.003792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:46.003737    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44" podStartSLOduration=2.003718887 podStartE2EDuration="2.003718887s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:43:46.000191391 +0000 UTC m=+2436.260429646" watchObservedRunningTime="2026-10-30 13:43:46.003718887 +0000 UTC m=+2436.263957121"
Oct 30 13:43:46.968792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:46.968751    4291 generic.go:356] "Generic (PLEG): container finished" podID="fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" containerID="667874c1904adfba568a707c57add78f323de20bc98518da4cfa92cd117149e1" exitCode=0
Oct 30 13:43:46.969218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:46.968843    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44" event={"ID":"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e","Type":"ContainerDied","Data":"667874c1904adfba568a707c57add78f323de20bc98518da4cfa92cd117149e1"}
Oct 30 13:43:46.970590 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:46.970565    4291 generic.go:356] "Generic (PLEG): container finished" podID="8a6d1b0f-2965-41ec-a32e-4a688d8542ef" containerID="fb0d3bffec6af36006b60961a2943f963da1e39215aba465be6801c326a66622" exitCode=0
Oct 30 13:43:46.970721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:46.970618    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v" event={"ID":"8a6d1b0f-2965-41ec-a32e-4a688d8542ef","Type":"ContainerDied","Data":"fb0d3bffec6af36006b60961a2943f963da1e39215aba465be6801c326a66622"}
Oct 30 13:43:48.157017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.156987    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:48.193709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.193683    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:48.245789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.245695    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-xks-ca\") pod \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") "
Oct 30 13:43:48.245789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.245783    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-kube-api-access-p9v4b\" (UniqueName: \"kubernetes.io/projected/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-kube-api-access-p9v4b\") pod \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") "
Oct 30 13:43:48.245994 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.245854    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-openshift-ca\") pod \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\" (UID: \"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e\") "
Oct 30 13:43:48.246965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.246937    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-openshift-ca" pod "fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" (UID: "fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:48.249808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.249780    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-kube-api-access-p9v4b" pod "fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" (UID: "fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e"). InnerVolumeSpecName "kube-api-access-p9v4b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:48.249931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.249859    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-xks-ca" pod "fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" (UID: "fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:48.347019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.346981    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-kube-api-access-lm85s\" (UniqueName: \"kubernetes.io/projected/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-kube-api-access-lm85s\") pod \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") "
Oct 30 13:43:48.347019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.347024    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-openshift-ca\") pod \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") "
Oct 30 13:43:48.347301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.347129    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-xks-ca\") pod \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\" (UID: \"8a6d1b0f-2965-41ec-a32e-4a688d8542ef\") "
Oct 30 13:43:48.347483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.347455    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-openshift-ca" pod "8a6d1b0f-2965-41ec-a32e-4a688d8542ef" (UID: "8a6d1b0f-2965-41ec-a32e-4a688d8542ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:43:48.347877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.347857    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.347960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.347882    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9v4b\" (UniqueName: \"kubernetes.io/projected/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-kube-api-access-p9v4b\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.347960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.347896    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.347960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.347910    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.349379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.349352    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-xks-ca" pod "8a6d1b0f-2965-41ec-a32e-4a688d8542ef" (UID: "8a6d1b0f-2965-41ec-a32e-4a688d8542ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:43:48.349449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.349413    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-kube-api-access-lm85s" pod "8a6d1b0f-2965-41ec-a32e-4a688d8542ef" (UID: "8a6d1b0f-2965-41ec-a32e-4a688d8542ef"). InnerVolumeSpecName "kube-api-access-lm85s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:43:48.449258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.449220    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm85s\" (UniqueName: \"kubernetes.io/projected/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-kube-api-access-lm85s\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.449258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.449254    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8a6d1b0f-2965-41ec-a32e-4a688d8542ef-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:43:48.986413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.986382    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"
Oct 30 13:43:48.986613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.986418    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44" event={"ID":"fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e","Type":"ContainerDied","Data":"ac50c6ab5d9744998d16bb4cdd7da4cc2f17f621999f30e8312fbbee28d091e8"}
Oct 30 13:43:48.986613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.986480    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac50c6ab5d9744998d16bb4cdd7da4cc2f17f621999f30e8312fbbee28d091e8"
Oct 30 13:43:48.987978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.987953    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v" event={"ID":"8a6d1b0f-2965-41ec-a32e-4a688d8542ef","Type":"ContainerDied","Data":"7792bc1d6f0e698706fa92ea5e7de2a5b314382435faaca44032e12e0d28f85a"}
Oct 30 13:43:48.988115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.987985    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"
Oct 30 13:43:48.988115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:48.987998    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7792bc1d6f0e698706fa92ea5e7de2a5b314382435faaca44032e12e0d28f85a"
Oct 30 13:43:56.275073 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:56.275033    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"]
Oct 30 13:43:57.303283 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.303240    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"]
Oct 30 13:43:57.303940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.303924    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8a6d1b0f-2965-41ec-a32e-4a688d8542ef" containerName="revoke-keys"
Oct 30 13:43:57.303940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.303940    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a6d1b0f-2965-41ec-a32e-4a688d8542ef" containerName="revoke-keys"
Oct 30 13:43:57.304026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.303954    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" containerName="revoke-keys"
Oct 30 13:43:57.304026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.303959    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" containerName="revoke-keys"
Oct 30 13:43:57.304095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.304063    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a6d1b0f-2965-41ec-a32e-4a688d8542ef" containerName="revoke-keys"
Oct 30 13:43:57.304095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.304072    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" containerName="revoke-keys"
Oct 30 13:43:57.308918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.308893    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.329784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.329749    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:43:57.341724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.341686    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:43:57.342045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.342017    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:43:57.350206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.350171    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"]
Oct 30 13:43:57.354302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.354240    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:43:57.400878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.400833    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.401069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.401025    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.401204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.401187    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pwff\" (UniqueName: \"kubernetes.io/projected/040eee6b-0b4b-4651-b56f-81edd16b4e2c-kube-api-access-9pwff\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.502682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.502597    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.502890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.502729    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.502890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.502819    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pwff\" (UniqueName: \"kubernetes.io/projected/040eee6b-0b4b-4651-b56f-81edd16b4e2c-kube-api-access-9pwff\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.503498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.503469    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.505120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.505098    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.527470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.527435    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pwff\" (UniqueName: \"kubernetes.io/projected/040eee6b-0b4b-4651-b56f-81edd16b4e2c-kube-api-access-9pwff\") pod \"maas-api-revoke-sub-4d6bbd5f-kswrs\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.623849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.623816    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:43:57.946508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:57.946477    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"]
Oct 30 13:43:57.950305 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:43:57.950269    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod040eee6b_0b4b_4651_b56f_81edd16b4e2c.slice/crio-d07d1aff6fe689468a984d2d2e563dfab42b9e7af06ac79d83c3e9bed979d235 WatchSource:0}: Error finding container d07d1aff6fe689468a984d2d2e563dfab42b9e7af06ac79d83c3e9bed979d235: Status 404 returned error can't find the container with id d07d1aff6fe689468a984d2d2e563dfab42b9e7af06ac79d83c3e9bed979d235
Oct 30 13:43:58.057170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:58.057125    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs" event={"ID":"040eee6b-0b4b-4651-b56f-81edd16b4e2c","Type":"ContainerStarted","Data":"d07d1aff6fe689468a984d2d2e563dfab42b9e7af06ac79d83c3e9bed979d235"}
Oct 30 13:43:58.961667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:58.947735    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"]
Oct 30 13:43:58.968663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:58.963124    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.025959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.024720    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"]
Oct 30 13:43:59.172134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.169700    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.172134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.169886    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hg8t2\" (UniqueName: \"kubernetes.io/projected/9e9a2737-debf-434c-9eda-7408d24906f3-kube-api-access-hg8t2\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.172134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.170054    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.183019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.182975    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs" event={"ID":"040eee6b-0b4b-4651-b56f-81edd16b4e2c","Type":"ContainerStarted","Data":"f4de12dcd85b610f272aa0a049c689c503cbb87dd55dcc2c621086e5b4928f7a"}
Oct 30 13:43:59.271626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.271502    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hg8t2\" (UniqueName: \"kubernetes.io/projected/9e9a2737-debf-434c-9eda-7408d24906f3-kube-api-access-hg8t2\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.271871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.271698    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.271871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.271771    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.272902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.272872    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.275252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.275141    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.294298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.294265    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hg8t2\" (UniqueName: \"kubernetes.io/projected/9e9a2737-debf-434c-9eda-7408d24906f3-kube-api-access-hg8t2\") pod \"maas-api-revoke-sub-72d40826-rc45p\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.361193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.361142    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:43:59.713951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:43:59.713920    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"]
Oct 30 13:43:59.714792 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:43:59.714767    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9a2737_debf_434c_9eda_7408d24906f3.slice/crio-37cefe5d64ca3a1a4f826e2abe51d3a2b73c57817fa8ec56b7b1c4e4c0e5f3c1 WatchSource:0}: Error finding container 37cefe5d64ca3a1a4f826e2abe51d3a2b73c57817fa8ec56b7b1c4e4c0e5f3c1: Status 404 returned error can't find the container with id 37cefe5d64ca3a1a4f826e2abe51d3a2b73c57817fa8ec56b7b1c4e4c0e5f3c1
Oct 30 13:44:00.194483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.194448    4291 generic.go:356] "Generic (PLEG): container finished" podID="040eee6b-0b4b-4651-b56f-81edd16b4e2c" containerID="f4de12dcd85b610f272aa0a049c689c503cbb87dd55dcc2c621086e5b4928f7a" exitCode=0
Oct 30 13:44:00.194944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.194553    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs" event={"ID":"040eee6b-0b4b-4651-b56f-81edd16b4e2c","Type":"ContainerDied","Data":"f4de12dcd85b610f272aa0a049c689c503cbb87dd55dcc2c621086e5b4928f7a"}
Oct 30 13:44:00.197251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.197220    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p" event={"ID":"9e9a2737-debf-434c-9eda-7408d24906f3","Type":"ContainerStarted","Data":"4b630079fac98b91975839bf087f5a728cc79a82cdf24b461583997e48aa7172"}
Oct 30 13:44:00.197379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.197279    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p" event={"ID":"9e9a2737-debf-434c-9eda-7408d24906f3","Type":"ContainerStarted","Data":"37cefe5d64ca3a1a4f826e2abe51d3a2b73c57817fa8ec56b7b1c4e4c0e5f3c1"}
Oct 30 13:44:00.200289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.200262    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"]
Oct 30 13:44:00.205624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.205594    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.230396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.230355    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"]
Oct 30 13:44:00.275312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.272530    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p" podStartSLOduration=2.272508372 podStartE2EDuration="2.272508372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:43:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:00.270119208 +0000 UTC m=+2450.530357443" watchObservedRunningTime="2026-10-30 13:44:00.272508372 +0000 UTC m=+2450.532746609"
Oct 30 13:44:00.283764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.283709    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.283908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.283803    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7xz\" (UniqueName: \"kubernetes.io/projected/619a17f0-c72d-4071-9daf-d15e69524416-kube-api-access-9c7xz\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.284688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.284026    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.385490 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.385445    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.385717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.385559    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7xz\" (UniqueName: \"kubernetes.io/projected/619a17f0-c72d-4071-9daf-d15e69524416-kube-api-access-9c7xz\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.385800 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.385720    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.386367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.386347    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.387956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.387930    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.405105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.405067    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7xz\" (UniqueName: \"kubernetes.io/projected/619a17f0-c72d-4071-9daf-d15e69524416-kube-api-access-9c7xz\") pod \"maas-api-revoke-sub-2c83715b-hfwdq\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.427378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.427351    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:44:00.535657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.529718    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:00.549967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.549931    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/040eee6b-0b4b-4651-b56f-81edd16b4e2c-kube-api-access-9pwff\" (UniqueName: \"kubernetes.io/projected/040eee6b-0b4b-4651-b56f-81edd16b4e2c-kube-api-access-9pwff\") pod \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") "
Oct 30 13:44:00.579817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.572943    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-openshift-ca\") pod \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") "
Oct 30 13:44:00.579817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.573033    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-xks-ca\") pod \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\" (UID: \"040eee6b-0b4b-4651-b56f-81edd16b4e2c\") "
Oct 30 13:44:00.582413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.582333    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-openshift-ca" pod "040eee6b-0b4b-4651-b56f-81edd16b4e2c" (UID: "040eee6b-0b4b-4651-b56f-81edd16b4e2c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:00.679937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.679749    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:00.687930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.686009    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/040eee6b-0b4b-4651-b56f-81edd16b4e2c-kube-api-access-9pwff" pod "040eee6b-0b4b-4651-b56f-81edd16b4e2c" (UID: "040eee6b-0b4b-4651-b56f-81edd16b4e2c"). InnerVolumeSpecName "kube-api-access-9pwff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:00.687930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.686129    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-xks-ca" pod "040eee6b-0b4b-4651-b56f-81edd16b4e2c" (UID: "040eee6b-0b4b-4651-b56f-81edd16b4e2c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:00.781945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.781608    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/040eee6b-0b4b-4651-b56f-81edd16b4e2c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:00.781945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:00.781682    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pwff\" (UniqueName: \"kubernetes.io/projected/040eee6b-0b4b-4651-b56f-81edd16b4e2c-kube-api-access-9pwff\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:01.050268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.050236    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"]
Oct 30 13:44:01.050812 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:44:01.050785    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619a17f0_c72d_4071_9daf_d15e69524416.slice/crio-87b20121b3a71d42e4a1280c587edc358f65a9597f312feb87dfab8c49c69146 WatchSource:0}: Error finding container 87b20121b3a71d42e4a1280c587edc358f65a9597f312feb87dfab8c49c69146: Status 404 returned error can't find the container with id 87b20121b3a71d42e4a1280c587edc358f65a9597f312feb87dfab8c49c69146
Oct 30 13:44:01.206047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.206008    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq" event={"ID":"619a17f0-c72d-4071-9daf-d15e69524416","Type":"ContainerStarted","Data":"87b20121b3a71d42e4a1280c587edc358f65a9597f312feb87dfab8c49c69146"}
Oct 30 13:44:01.207563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.207535    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"
Oct 30 13:44:01.207712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.207538    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs" event={"ID":"040eee6b-0b4b-4651-b56f-81edd16b4e2c","Type":"ContainerDied","Data":"d07d1aff6fe689468a984d2d2e563dfab42b9e7af06ac79d83c3e9bed979d235"}
Oct 30 13:44:01.207712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.207663    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d07d1aff6fe689468a984d2d2e563dfab42b9e7af06ac79d83c3e9bed979d235"
Oct 30 13:44:01.209272 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.209246    4291 generic.go:356] "Generic (PLEG): container finished" podID="9e9a2737-debf-434c-9eda-7408d24906f3" containerID="4b630079fac98b91975839bf087f5a728cc79a82cdf24b461583997e48aa7172" exitCode=0
Oct 30 13:44:01.209411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.209296    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p" event={"ID":"9e9a2737-debf-434c-9eda-7408d24906f3","Type":"ContainerDied","Data":"4b630079fac98b91975839bf087f5a728cc79a82cdf24b461583997e48aa7172"}
Oct 30 13:44:01.275934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.275884    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x" podUID="4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" containerName="payload-processing" containerID="cri-o://fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e" gracePeriod=25
Oct 30 13:44:01.572301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.572221    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:44:01.701841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.701801    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-routing-overlay-volume\") pod \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") "
Oct 30 13:44:01.702082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.701881    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-kube-api-access-4nw4n\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-kube-api-access-4nw4n\") pod \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") "
Oct 30 13:44:01.702082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.701979    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-plugins-config-volume\") pod \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") "
Oct 30 13:44:01.702217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.702095    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-provider-credentials\") pod \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\" (UID: \"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c\") "
Oct 30 13:44:01.702371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.702231    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-routing-overlay-volume" pod "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" (UID: "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:01.702470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.702429    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-plugins-config-volume" pod "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" (UID: "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:01.702799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.702773    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:01.702899 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.702801    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:01.704549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.704520    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-provider-credentials" pod "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" (UID: "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:01.704714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.704589    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-kube-api-access-4nw4n" pod "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" (UID: "4def8600-3f7a-4c69-a6dc-9fefdc97cf3c"). InnerVolumeSpecName "kube-api-access-4nw4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:01.803792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.803759    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:01.803792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:01.803787    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nw4n\" (UniqueName: \"kubernetes.io/projected/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c-kube-api-access-4nw4n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:02.218442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.218327    4291 generic.go:356] "Generic (PLEG): container finished" podID="4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" containerID="fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e" exitCode=0
Oct 30 13:44:02.218442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.218397    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"
Oct 30 13:44:02.218442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.218405    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x" event={"ID":"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c","Type":"ContainerDied","Data":"fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e"}
Oct 30 13:44:02.219025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.218492    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x" event={"ID":"4def8600-3f7a-4c69-a6dc-9fefdc97cf3c","Type":"ContainerDied","Data":"4364395b0755135dd0f3373589d18f1bc540641af29c1e8f38dad61a736eb0d0"}
Oct 30 13:44:02.219025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.218522    4291 scope.go:122] "RemoveContainer" containerID="fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e"
Oct 30 13:44:02.220592 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.220561    4291 generic.go:356] "Generic (PLEG): container finished" podID="619a17f0-c72d-4071-9daf-d15e69524416" containerID="9aa84cc4e0ccbeef8ceb30777cd447216b3eca8297a5d5114a7a433f161d6dd7" exitCode=0
Oct 30 13:44:02.220773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.220686    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq" event={"ID":"619a17f0-c72d-4071-9daf-d15e69524416","Type":"ContainerDied","Data":"9aa84cc4e0ccbeef8ceb30777cd447216b3eca8297a5d5114a7a433f161d6dd7"}
Oct 30 13:44:02.229091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.229069    4291 scope.go:122] "RemoveContainer" containerID="fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e"
Oct 30 13:44:02.229483 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:44:02.229456    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e\": container with ID starting with fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e not found: ID does not exist" containerID="fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e"
Oct 30 13:44:02.229599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.229493    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e"} err="failed to get container status \"fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e\": rpc error: code = NotFound desc = could not find container \"fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e\": container with ID starting with fd0979e6d00c3d16f4363b1e10f8d29a913538336fa87b7b13c298dadf8b3d6e not found: ID does not exist"
Oct 30 13:44:02.280612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.280575    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"]
Oct 30 13:44:02.292430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.292389    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-84f76bj5x"]
Oct 30 13:44:02.381992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.381965    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:44:02.513896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.513806    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-openshift-ca\") pod \"9e9a2737-debf-434c-9eda-7408d24906f3\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") "
Oct 30 13:44:02.513896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.513853    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-xks-ca\") pod \"9e9a2737-debf-434c-9eda-7408d24906f3\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") "
Oct 30 13:44:02.514156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.514134    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9a2737-debf-434c-9eda-7408d24906f3-kube-api-access-hg8t2\" (UniqueName: \"kubernetes.io/projected/9e9a2737-debf-434c-9eda-7408d24906f3-kube-api-access-hg8t2\") pod \"9e9a2737-debf-434c-9eda-7408d24906f3\" (UID: \"9e9a2737-debf-434c-9eda-7408d24906f3\") "
Oct 30 13:44:02.514257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.514238    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-openshift-ca" pod "9e9a2737-debf-434c-9eda-7408d24906f3" (UID: "9e9a2737-debf-434c-9eda-7408d24906f3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:02.515031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.514690    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:02.516389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.516362    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-xks-ca" pod "9e9a2737-debf-434c-9eda-7408d24906f3" (UID: "9e9a2737-debf-434c-9eda-7408d24906f3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:02.516508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.516386    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9a2737-debf-434c-9eda-7408d24906f3-kube-api-access-hg8t2" pod "9e9a2737-debf-434c-9eda-7408d24906f3" (UID: "9e9a2737-debf-434c-9eda-7408d24906f3"). InnerVolumeSpecName "kube-api-access-hg8t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:02.616625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.616590    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hg8t2\" (UniqueName: \"kubernetes.io/projected/9e9a2737-debf-434c-9eda-7408d24906f3-kube-api-access-hg8t2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:02.616625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:02.616621    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e9a2737-debf-434c-9eda-7408d24906f3-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:03.230557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.230471    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"
Oct 30 13:44:03.230557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.230466    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p" event={"ID":"9e9a2737-debf-434c-9eda-7408d24906f3","Type":"ContainerDied","Data":"37cefe5d64ca3a1a4f826e2abe51d3a2b73c57817fa8ec56b7b1c4e4c0e5f3c1"}
Oct 30 13:44:03.230557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.230531    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37cefe5d64ca3a1a4f826e2abe51d3a2b73c57817fa8ec56b7b1c4e4c0e5f3c1"
Oct 30 13:44:03.405066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.405017    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:03.529007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.528893    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-xks-ca\") pod \"619a17f0-c72d-4071-9daf-d15e69524416\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") "
Oct 30 13:44:03.529215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.529035    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619a17f0-c72d-4071-9daf-d15e69524416-kube-api-access-9c7xz\" (UniqueName: \"kubernetes.io/projected/619a17f0-c72d-4071-9daf-d15e69524416-kube-api-access-9c7xz\") pod \"619a17f0-c72d-4071-9daf-d15e69524416\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") "
Oct 30 13:44:03.529215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.529114    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-openshift-ca\") pod \"619a17f0-c72d-4071-9daf-d15e69524416\" (UID: \"619a17f0-c72d-4071-9daf-d15e69524416\") "
Oct 30 13:44:03.529516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.529478    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-openshift-ca" pod "619a17f0-c72d-4071-9daf-d15e69524416" (UID: "619a17f0-c72d-4071-9daf-d15e69524416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:03.531163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.531136    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-xks-ca" pod "619a17f0-c72d-4071-9daf-d15e69524416" (UID: "619a17f0-c72d-4071-9daf-d15e69524416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:03.531278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.531260    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619a17f0-c72d-4071-9daf-d15e69524416-kube-api-access-9c7xz" pod "619a17f0-c72d-4071-9daf-d15e69524416" (UID: "619a17f0-c72d-4071-9daf-d15e69524416"). InnerVolumeSpecName "kube-api-access-9c7xz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:03.631298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.631253    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7xz\" (UniqueName: \"kubernetes.io/projected/619a17f0-c72d-4071-9daf-d15e69524416-kube-api-access-9c7xz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:03.631298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.631287    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:03.631298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.631298    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/619a17f0-c72d-4071-9daf-d15e69524416-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:03.879048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:03.879003    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" path="/var/lib/kubelet/pods/4def8600-3f7a-4c69-a6dc-9fefdc97cf3c/volumes"
Oct 30 13:44:04.239094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:04.239004    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"
Oct 30 13:44:04.239094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:04.239041    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq" event={"ID":"619a17f0-c72d-4071-9daf-d15e69524416","Type":"ContainerDied","Data":"87b20121b3a71d42e4a1280c587edc358f65a9597f312feb87dfab8c49c69146"}
Oct 30 13:44:04.239559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:04.239100    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87b20121b3a71d42e4a1280c587edc358f65a9597f312feb87dfab8c49c69146"
Oct 30 13:44:08.213118 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:44:08.213070    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:44:14.365695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.365660    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"]
Oct 30 13:44:14.366298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366282    4291 cpu_manager.go:414] "containerMap: removing container" podUID="040eee6b-0b4b-4651-b56f-81edd16b4e2c" containerName="revoke-keys"
Oct 30 13:44:14.366346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366299    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="040eee6b-0b4b-4651-b56f-81edd16b4e2c" containerName="revoke-keys"
Oct 30 13:44:14.366346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366311    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9e9a2737-debf-434c-9eda-7408d24906f3" containerName="revoke-keys"
Oct 30 13:44:14.366346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366317    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9a2737-debf-434c-9eda-7408d24906f3" containerName="revoke-keys"
Oct 30 13:44:14.366346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366338    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" containerName="payload-processing"
Oct 30 13:44:14.366346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366343    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" containerName="payload-processing"
Oct 30 13:44:14.366512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366353    4291 cpu_manager.go:414] "containerMap: removing container" podUID="619a17f0-c72d-4071-9daf-d15e69524416" containerName="revoke-keys"
Oct 30 13:44:14.366512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366358    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619a17f0-c72d-4071-9daf-d15e69524416" containerName="revoke-keys"
Oct 30 13:44:14.366512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366448    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4def8600-3f7a-4c69-a6dc-9fefdc97cf3c" containerName="payload-processing"
Oct 30 13:44:14.366512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366458    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="619a17f0-c72d-4071-9daf-d15e69524416" containerName="revoke-keys"
Oct 30 13:44:14.366512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366469    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="040eee6b-0b4b-4651-b56f-81edd16b4e2c" containerName="revoke-keys"
Oct 30 13:44:14.366512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.366475    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9a2737-debf-434c-9eda-7408d24906f3" containerName="revoke-keys"
Oct 30 13:44:14.371246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.371224    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.410863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.410831    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"agc-cred-2f94a7608e69\""
Oct 30 13:44:14.411082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.411060    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-plugins-e2e-extmod-a-310546\""
Oct 30 13:44:14.411183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.411167    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-e2e--175bf124-dockercfg-25knt\""
Oct 30 13:44:14.411337 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.411320    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"openshift-service-ca.crt\""
Oct 30 13:44:14.411416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.411402    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"kube-root-ca.crt\""
Oct 30 13:44:14.411476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.411404    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"routing-overlay\""
Oct 30 13:44:14.415924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.415896    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"]
Oct 30 13:44:14.423724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.423693    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n9pcj\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-kube-api-access-n9pcj\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.423916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.423811    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.423916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.423857    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.423916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.423882    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.527700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.527650    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.528494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.528465    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.529102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.529078    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n9pcj\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-kube-api-access-n9pcj\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.529353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.529331    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.555670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.535280    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.591782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.537379    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.601790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.556598    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.602030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.598533    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n9pcj\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-kube-api-access-n9pcj\") pod \"payload-processing-external-model-e2e-extmod-a-310546-597f78ch4\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:14.688284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:14.686986    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:15.092176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.090584    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"]
Oct 30 13:44:15.375463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.375428    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4" event={"ID":"291e6890-c751-4659-b268-239d2c9056e9","Type":"ContainerStarted","Data":"0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f"}
Oct 30 13:44:15.375941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.375487    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4" event={"ID":"291e6890-c751-4659-b268-239d2c9056e9","Type":"ContainerStarted","Data":"1d027c3af7cae9a73a1a4d0620dc3277b4d42dcc331e5c7e7fbc63435f9b5aa0"}
Oct 30 13:44:15.375941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.375579    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:15.427600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.427543    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4" podStartSLOduration=1.427528024 podStartE2EDuration="1.427528024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:15.426584056 +0000 UTC m=+2465.686822290" watchObservedRunningTime="2026-10-30 13:44:15.427528024 +0000 UTC m=+2465.687766258"
Oct 30 13:44:15.688017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.687930    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"]
Oct 30 13:44:15.692578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.692547    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.713069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.713036    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"]
Oct 30 13:44:15.784925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.784881    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkr5\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-kube-api-access-wjkr5\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.784996    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.785069    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.785150    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.785185    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.785207    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.785318    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.785366    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.785420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.785387    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886302    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886352    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886377    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886430    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886484    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886519    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886559    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkr5\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-kube-api-access-wjkr5\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886622    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886728    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.886948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.886804    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.887133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.887112    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.887313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.887274    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.887313 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.887306    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.887752 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.887719    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.889282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.889253    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.889282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.889269    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.912981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.912952    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:15.918469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:15.918436    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkr5\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-kube-api-access-wjkr5\") pod \"maas-default-gateway-openshift-default-67bf986f99-qng5l\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:16.010442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:16.010351    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:16.336954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:16.336910    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"]
Oct 30 13:44:16.337759 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:44:16.337718    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77f16eb_16b5_40cb_a1ac_2f0cc8da1801.slice/crio-e13d48e665803c4723157bf937370d4948e35dd1a0be6eff171b1914e5111db3 WatchSource:0}: Error finding container e13d48e665803c4723157bf937370d4948e35dd1a0be6eff171b1914e5111db3: Status 404 returned error can't find the container with id e13d48e665803c4723157bf937370d4948e35dd1a0be6eff171b1914e5111db3
Oct 30 13:44:16.340977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:16.340940    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:44:16.341063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:16.341011    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:44:16.341063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:16.341040    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:44:16.385598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:16.385565    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" event={"ID":"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801","Type":"ContainerStarted","Data":"e13d48e665803c4723157bf937370d4948e35dd1a0be6eff171b1914e5111db3"}
Oct 30 13:44:17.443673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:17.442172    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" event={"ID":"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801","Type":"ContainerStarted","Data":"278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102"}
Oct 30 13:44:17.531656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:17.530499    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podStartSLOduration=2.530476637 podStartE2EDuration="2.530476637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:17.513006331 +0000 UTC m=+2467.773244566" watchObservedRunningTime="2026-10-30 13:44:17.530476637 +0000 UTC m=+2467.790714874"
Oct 30 13:44:18.010621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:18.010585    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:18.011952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:18.011921    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qng5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:44:18.012125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:18.011979    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:44:19.011230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:19.011185    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qng5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:44:19.011725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:19.011260    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:44:20.011663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:20.011605    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qng5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:44:20.012213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:20.011680    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:44:21.015745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:21.012213    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qng5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:44:21.015745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:21.012344    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:44:22.011221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:22.011184    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qng5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:44:22.011417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:22.011240    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:44:23.011493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:23.011394    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qng5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:44:23.011493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:23.011463    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:44:24.011040 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:24.010997    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qng5l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused" start-of-body=
Oct 30 13:44:24.011246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:24.011074    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.234:15021/healthz/ready\": dial tcp 10.133.0.234:15021: connect: connection refused"
Oct 30 13:44:25.211716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:25.211673    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:25.212901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:25.212829    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:25.214435 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:25.214411    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:44:26.390704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:26.390600    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:44:30.315705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:30.315661    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:44:34.208894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:34.208851    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:44:34.209552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:34.209518    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:44:37.981852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:37.981815    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"]
Oct 30 13:44:37.986251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:37.986225    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.003666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.003607    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:44:38.015486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.015454    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:44:38.020486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.020453    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"]
Oct 30 13:44:38.092136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.092104    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:38.092366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.092213    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:44:38.142648 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.142591    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.142849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.142811    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.143022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.143000    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgfcb\" (UniqueName: \"kubernetes.io/projected/38d8474a-3aa9-4490-9f48-42af329743b7-kube-api-access-fgfcb\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.246594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.246463    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.247712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.247683    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgfcb\" (UniqueName: \"kubernetes.io/projected/38d8474a-3aa9-4490-9f48-42af329743b7-kube-api-access-fgfcb\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.248331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.248307    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.249349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.249317    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.253362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.253335    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.296271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.293045    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgfcb\" (UniqueName: \"kubernetes.io/projected/38d8474a-3aa9-4490-9f48-42af329743b7-kube-api-access-fgfcb\") pod \"maas-api-revoke-sub-2d74f52f-jg4pb\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.296271 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:44:38.294649    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:44:38.301916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.301872    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:38.702499 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:38.702466    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"]
Oct 30 13:44:38.704165 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:44:38.704137    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38d8474a_3aa9_4490_9f48_42af329743b7.slice/crio-a8c0cc397ed2c409373ea360ae0c4ad7f52d8575a5f3f4f3530105ffc3a65744 WatchSource:0}: Error finding container a8c0cc397ed2c409373ea360ae0c4ad7f52d8575a5f3f4f3530105ffc3a65744: Status 404 returned error can't find the container with id a8c0cc397ed2c409373ea360ae0c4ad7f52d8575a5f3f4f3530105ffc3a65744
Oct 30 13:44:39.069072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.069033    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb" event={"ID":"38d8474a-3aa9-4490-9f48-42af329743b7","Type":"ContainerStarted","Data":"a8c0cc397ed2c409373ea360ae0c4ad7f52d8575a5f3f4f3530105ffc3a65744"}
Oct 30 13:44:39.330215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.330122    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"]
Oct 30 13:44:39.334687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.334661    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.355362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.355328    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"agc-cred-a803a075119b\""
Oct 30 13:44:39.356175 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.356137    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"payload-processing-external-model-e2e--2f6d7ae8-dockercfg-b2ggj\""
Oct 30 13:44:39.368006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.367979    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c3efb2\""
Oct 30 13:44:39.368006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.367979    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"routing-overlay\""
Oct 30 13:44:39.368186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.367984    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"kube-root-ca.crt\""
Oct 30 13:44:39.368186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.368123    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"openshift-service-ca.crt\""
Oct 30 13:44:39.377690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.377658    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"]
Oct 30 13:44:39.383628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.383589    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.383628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.383655    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.383902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.383782    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5d5wn\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-kube-api-access-5d5wn\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.383968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.383937    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.561707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.551369    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.561707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.551717    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.561707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.551748    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.561707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.551788    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5d5wn\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-kube-api-access-5d5wn\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.570726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.565068    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.570969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.566397    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.578790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.578668    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5d5wn\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-kube-api-access-5d5wn\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.585734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.585661    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.650969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.650937    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:39.972717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:39.972689    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"]
Oct 30 13:44:39.973579 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:44:39.973553    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc309c7_1d98_49f1_a0bd_e22b351c1178.slice/crio-8693ba91737b83c0073c2b77f8cf6089e7f15d96efc84771fdc48922b19671a0 WatchSource:0}: Error finding container 8693ba91737b83c0073c2b77f8cf6089e7f15d96efc84771fdc48922b19671a0: Status 404 returned error can't find the container with id 8693ba91737b83c0073c2b77f8cf6089e7f15d96efc84771fdc48922b19671a0
Oct 30 13:44:40.078124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:40.078089    4291 generic.go:356] "Generic (PLEG): container finished" podID="38d8474a-3aa9-4490-9f48-42af329743b7" containerID="0f3bd4e9069d3ca93d8e529ed209b70d7bf5640eea3c2ff6454bbaf610efae23" exitCode=0
Oct 30 13:44:40.078941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:40.078173    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb" event={"ID":"38d8474a-3aa9-4490-9f48-42af329743b7","Type":"ContainerDied","Data":"0f3bd4e9069d3ca93d8e529ed209b70d7bf5640eea3c2ff6454bbaf610efae23"}
Oct 30 13:44:40.080285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:40.080240    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws" event={"ID":"7dc309c7-1d98-49f1-a0bd-e22b351c1178","Type":"ContainerStarted","Data":"2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d"}
Oct 30 13:44:40.080438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:40.080295    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws" event={"ID":"7dc309c7-1d98-49f1-a0bd-e22b351c1178","Type":"ContainerStarted","Data":"8693ba91737b83c0073c2b77f8cf6089e7f15d96efc84771fdc48922b19671a0"}
Oct 30 13:44:40.080438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:40.080341    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:40.182144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:40.182072    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws" podStartSLOduration=1.18205144 podStartE2EDuration="1.18205144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:40.181224823 +0000 UTC m=+2490.441463057" watchObservedRunningTime="2026-10-30 13:44:40.18205144 +0000 UTC m=+2490.442289671"
Oct 30 13:44:41.267672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.267628    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:41.374201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.374154    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-xks-ca\") pod \"38d8474a-3aa9-4490-9f48-42af329743b7\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") "
Oct 30 13:44:41.374395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.374210    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-openshift-ca\") pod \"38d8474a-3aa9-4490-9f48-42af329743b7\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") "
Oct 30 13:44:41.374395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.374270    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38d8474a-3aa9-4490-9f48-42af329743b7-kube-api-access-fgfcb\" (UniqueName: \"kubernetes.io/projected/38d8474a-3aa9-4490-9f48-42af329743b7-kube-api-access-fgfcb\") pod \"38d8474a-3aa9-4490-9f48-42af329743b7\" (UID: \"38d8474a-3aa9-4490-9f48-42af329743b7\") "
Oct 30 13:44:41.374734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.374703    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-openshift-ca" pod "38d8474a-3aa9-4490-9f48-42af329743b7" (UID: "38d8474a-3aa9-4490-9f48-42af329743b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:41.374959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.374936    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:41.377023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.376995    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38d8474a-3aa9-4490-9f48-42af329743b7-kube-api-access-fgfcb" pod "38d8474a-3aa9-4490-9f48-42af329743b7" (UID: "38d8474a-3aa9-4490-9f48-42af329743b7"). InnerVolumeSpecName "kube-api-access-fgfcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:41.377126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.376992    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-xks-ca" pod "38d8474a-3aa9-4490-9f48-42af329743b7" (UID: "38d8474a-3aa9-4490-9f48-42af329743b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:41.476061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.476022    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/38d8474a-3aa9-4490-9f48-42af329743b7-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:41.476061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:41.476055    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgfcb\" (UniqueName: \"kubernetes.io/projected/38d8474a-3aa9-4490-9f48-42af329743b7-kube-api-access-fgfcb\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:42.095116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:42.095081    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"
Oct 30 13:44:42.095116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:42.095102    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb" event={"ID":"38d8474a-3aa9-4490-9f48-42af329743b7","Type":"ContainerDied","Data":"a8c0cc397ed2c409373ea360ae0c4ad7f52d8575a5f3f4f3530105ffc3a65744"}
Oct 30 13:44:42.095353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:42.095173    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a8c0cc397ed2c409373ea360ae0c4ad7f52d8575a5f3f4f3530105ffc3a65744"
Oct 30 13:44:49.885656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.882349    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"]
Oct 30 13:44:49.885656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.883319    4291 cpu_manager.go:414] "containerMap: removing container" podUID="38d8474a-3aa9-4490-9f48-42af329743b7" containerName="revoke-keys"
Oct 30 13:44:49.885656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.883340    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38d8474a-3aa9-4490-9f48-42af329743b7" containerName="revoke-keys"
Oct 30 13:44:49.885656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.883499    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="38d8474a-3aa9-4490-9f48-42af329743b7" containerName="revoke-keys"
Oct 30 13:44:49.961777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.961746    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:49.973457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.973429    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"]
Oct 30 13:44:49.978259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.978232    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"e2e-cfl-e402e5-provider-cred\""
Oct 30 13:44:49.988989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.988944    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-b691dcc50cd7\""
Oct 30 13:44:49.989155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.989023    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-wk62z\""
Oct 30 13:44:49.989155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.989032    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\""
Oct 30 13:44:49.989155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.989089    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\""
Oct 30 13:44:49.989155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.989096    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\""
Oct 30 13:44:49.989155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:49.989135    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\""
Oct 30 13:44:50.104528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.104477    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxt6\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-kube-api-access-cgxt6\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.104769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.104556    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.104769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.104669    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.104769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.104720    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.145069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.144981    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"]
Oct 30 13:44:50.150069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.150046    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.172154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.172122    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:44:50.172689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.172350    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:44:50.173524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.173471    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:44:50.173664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.173586    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:44:50.177304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.177283    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"]
Oct 30 13:44:50.206096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.206052    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9snc9\" (UniqueName: \"kubernetes.io/projected/72412330-2213-4c10-9d32-8860d697c9eb-kube-api-access-9snc9\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.206372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.206249    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxt6\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-kube-api-access-cgxt6\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.206372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.206284    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.207003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.206468    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.207003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.206549    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.207985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.207707    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.207985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.207750    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.207985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.207759    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.207985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.207837    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.210580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.210416    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.235128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.235103    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxt6\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-kube-api-access-cgxt6\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.283416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.283377    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:50.309213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.309174    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9snc9\" (UniqueName: \"kubernetes.io/projected/72412330-2213-4c10-9d32-8860d697c9eb-kube-api-access-9snc9\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.309380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.309361    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.309523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.309506    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.310400 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.310375    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.312034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.312008    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.329405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.329374    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9snc9\" (UniqueName: \"kubernetes.io/projected/72412330-2213-4c10-9d32-8860d697c9eb-kube-api-access-9snc9\") pod \"maas-api-revoke-sub-a40bf410-5w4xv\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.464715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.464602    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:50.595577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.595542    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"]
Oct 30 13:44:50.596186 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:44:50.596161    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod998880a0_7ee4_452c_89ca_d2cc33a61c7a.slice/crio-f099afb158ba9ba2dff8ed16515d5e4191f31cebb11ded173bf50ae72eb3cddc WatchSource:0}: Error finding container f099afb158ba9ba2dff8ed16515d5e4191f31cebb11ded173bf50ae72eb3cddc: Status 404 returned error can't find the container with id f099afb158ba9ba2dff8ed16515d5e4191f31cebb11ded173bf50ae72eb3cddc
Oct 30 13:44:50.772758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:50.772722    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"]
Oct 30 13:44:50.772903 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:44:50.772815    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72412330_2213_4c10_9d32_8860d697c9eb.slice/crio-7bf36cd9f72be1556aef3dcbf7c8b6e5d4c7f87a17a1d5896286a35654307242 WatchSource:0}: Error finding container 7bf36cd9f72be1556aef3dcbf7c8b6e5d4c7f87a17a1d5896286a35654307242: Status 404 returned error can't find the container with id 7bf36cd9f72be1556aef3dcbf7c8b6e5d4c7f87a17a1d5896286a35654307242
Oct 30 13:44:51.098870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.098540    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:44:51.160656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.160590    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4" event={"ID":"998880a0-7ee4-452c-89ca-d2cc33a61c7a","Type":"ContainerStarted","Data":"c2d9f0634981e4d6f9221f4822a62628aabfea059c212d4e5d33090ab73c05a8"}
Oct 30 13:44:51.160876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.160855    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:44:51.160957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.160916    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4" event={"ID":"998880a0-7ee4-452c-89ca-d2cc33a61c7a","Type":"ContainerStarted","Data":"f099afb158ba9ba2dff8ed16515d5e4191f31cebb11ded173bf50ae72eb3cddc"}
Oct 30 13:44:51.162524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.162491    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv" event={"ID":"72412330-2213-4c10-9d32-8860d697c9eb","Type":"ContainerStarted","Data":"f7b9d27bb33453a507e6bab9c0680a7d1a1b0b802c7806907cfa5eec405b2510"}
Oct 30 13:44:51.162678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.162545    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv" event={"ID":"72412330-2213-4c10-9d32-8860d697c9eb","Type":"ContainerStarted","Data":"7bf36cd9f72be1556aef3dcbf7c8b6e5d4c7f87a17a1d5896286a35654307242"}
Oct 30 13:44:51.210826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.210764    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4" podStartSLOduration=2.210747205 podStartE2EDuration="2.210747205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:51.205564776 +0000 UTC m=+2501.465803010" watchObservedRunningTime="2026-10-30 13:44:51.210747205 +0000 UTC m=+2501.470985439"
Oct 30 13:44:51.255060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:51.255005    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv" podStartSLOduration=1.254986618 podStartE2EDuration="1.254986618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:44:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:44:51.250437566 +0000 UTC m=+2501.510675796" watchObservedRunningTime="2026-10-30 13:44:51.254986618 +0000 UTC m=+2501.515224852"
Oct 30 13:44:52.173487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:52.173453    4291 generic.go:356] "Generic (PLEG): container finished" podID="72412330-2213-4c10-9d32-8860d697c9eb" containerID="f7b9d27bb33453a507e6bab9c0680a7d1a1b0b802c7806907cfa5eec405b2510" exitCode=0
Oct 30 13:44:52.173991 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:52.173528    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv" event={"ID":"72412330-2213-4c10-9d32-8860d697c9eb","Type":"ContainerDied","Data":"f7b9d27bb33453a507e6bab9c0680a7d1a1b0b802c7806907cfa5eec405b2510"}
Oct 30 13:44:53.344071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.344047    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:53.459559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.459461    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-xks-ca\") pod \"72412330-2213-4c10-9d32-8860d697c9eb\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") "
Oct 30 13:44:53.459559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.459518    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72412330-2213-4c10-9d32-8860d697c9eb-kube-api-access-9snc9\" (UniqueName: \"kubernetes.io/projected/72412330-2213-4c10-9d32-8860d697c9eb-kube-api-access-9snc9\") pod \"72412330-2213-4c10-9d32-8860d697c9eb\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") "
Oct 30 13:44:53.459839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.459598    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-openshift-ca\") pod \"72412330-2213-4c10-9d32-8860d697c9eb\" (UID: \"72412330-2213-4c10-9d32-8860d697c9eb\") "
Oct 30 13:44:53.459934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.459909    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-openshift-ca" pod "72412330-2213-4c10-9d32-8860d697c9eb" (UID: "72412330-2213-4c10-9d32-8860d697c9eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:44:53.460282 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.460262    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:53.461666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.461616    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-xks-ca" pod "72412330-2213-4c10-9d32-8860d697c9eb" (UID: "72412330-2213-4c10-9d32-8860d697c9eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:44:53.461666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.461628    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72412330-2213-4c10-9d32-8860d697c9eb-kube-api-access-9snc9" pod "72412330-2213-4c10-9d32-8860d697c9eb" (UID: "72412330-2213-4c10-9d32-8860d697c9eb"). InnerVolumeSpecName "kube-api-access-9snc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:44:53.561993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.561957    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72412330-2213-4c10-9d32-8860d697c9eb-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:53.561993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:53.561986    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9snc9\" (UniqueName: \"kubernetes.io/projected/72412330-2213-4c10-9d32-8860d697c9eb-kube-api-access-9snc9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:44:54.189826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:54.189735    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"
Oct 30 13:44:54.189826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:54.189767    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv" event={"ID":"72412330-2213-4c10-9d32-8860d697c9eb","Type":"ContainerDied","Data":"7bf36cd9f72be1556aef3dcbf7c8b6e5d4c7f87a17a1d5896286a35654307242"}
Oct 30 13:44:54.189826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:44:54.189811    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bf36cd9f72be1556aef3dcbf7c8b6e5d4c7f87a17a1d5896286a35654307242"
Oct 30 13:45:00.186016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.185981    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"]
Oct 30 13:45:00.186701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.186660    4291 cpu_manager.go:414] "containerMap: removing container" podUID="72412330-2213-4c10-9d32-8860d697c9eb" containerName="revoke-keys"
Oct 30 13:45:00.186701 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.186679    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72412330-2213-4c10-9d32-8860d697c9eb" containerName="revoke-keys"
Oct 30 13:45:00.186962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.186863    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="72412330-2213-4c10-9d32-8860d697c9eb" containerName="revoke-keys"
Oct 30 13:45:00.190875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.190855    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"]
Oct 30 13:45:00.191008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.190974    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.195438 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.195420    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"]
Oct 30 13:45:00.195581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.195545    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.199675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.199652    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"]
Oct 30 13:45:00.199800 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.199707    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.204138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.204118    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"]
Oct 30 13:45:00.204244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.204223    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.208238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.208221    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.234382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.234361    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:45:00.234532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.234464    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:45:00.234532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.234470    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:45:00.251266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.251238    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"]
Oct 30 13:45:00.257437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.257402    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"]
Oct 30 13:45:00.262668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.262627    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"]
Oct 30 13:45:00.267250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.267225    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"]
Oct 30 13:45:00.275619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.275590    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"]
Oct 30 13:45:00.299296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.296848    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"]
Oct 30 13:45:00.307963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.307932    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.322855 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.322811    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"]
Oct 30 13:45:00.323035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.322945    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.338896    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"]
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.341286    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.350582    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"]
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.351496    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.352457    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrn6d\" (UniqueName: \"kubernetes.io/projected/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-kube-api-access-lrn6d\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353275    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353350    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tw6z8\" (UniqueName: \"kubernetes.io/projected/5714a3ae-1af2-43fd-824c-e6536556a50f-kube-api-access-tw6z8\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353398    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714a3ae-1af2-43fd-824c-e6536556a50f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353434    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32860ef1-4fcb-45f2-8297-f3330e738442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353607    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgf2z\" (UniqueName: \"kubernetes.io/projected/73b26525-0a60-471e-8fe4-2f5a9696b700-kube-api-access-zgf2z\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353704    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvfck\" (UniqueName: \"kubernetes.io/projected/5ec89a7d-eb27-48de-a793-86a7b0d5389b-kube-api-access-wvfck\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353743    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b26525-0a60-471e-8fe4-2f5a9696b700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.353968    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89a7d-eb27-48de-a793-86a7b0d5389b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.355656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.354050    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njxl4\" (UniqueName: \"kubernetes.io/projected/32860ef1-4fcb-45f2-8297-f3330e738442-kube-api-access-njxl4\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.359859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.359840    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.363381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.362986    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"]
Oct 30 13:45:00.373148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.373109    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"]
Oct 30 13:45:00.373275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.373197    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:00.373856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.373837    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"]
Oct 30 13:45:00.379216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.379198    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"]
Oct 30 13:45:00.382298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.380059    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:00.388918 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.388899    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"]
Oct 30 13:45:00.389153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.389139    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:00.391335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.391305    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"]
Oct 30 13:45:00.397154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.397134    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"]
Oct 30 13:45:00.402673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.402651    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"]
Oct 30 13:45:00.416742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.416722    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"]
Oct 30 13:45:00.422785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.422761    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"]
Oct 30 13:45:00.430780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.429840    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"]
Oct 30 13:45:00.430780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.429962    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.436822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.436763    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"]
Oct 30 13:45:00.475324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.475283    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696f6bbf-ec44-4546-8d71-29bd05cf026e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.478814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.478782    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrn6d\" (UniqueName: \"kubernetes.io/projected/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-kube-api-access-lrn6d\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.479091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.479065    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80678d19-5845-487b-9329-577e2cf9738e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.481593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481521    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bsrpp\" (UniqueName: \"kubernetes.io/projected/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-kube-api-access-bsrpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.481721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481624    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.481721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481681    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tw6z8\" (UniqueName: \"kubernetes.io/projected/5714a3ae-1af2-43fd-824c-e6536556a50f-kube-api-access-tw6z8\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.481721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481713    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714a3ae-1af2-43fd-824c-e6536556a50f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.481901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481745    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb4hk\" (UniqueName: \"kubernetes.io/projected/80678d19-5845-487b-9329-577e2cf9738e-kube-api-access-tb4hk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.481901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481832    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32860ef1-4fcb-45f2-8297-f3330e738442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.482015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481922    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgf2z\" (UniqueName: \"kubernetes.io/projected/73b26525-0a60-471e-8fe4-2f5a9696b700-kube-api-access-zgf2z\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.482015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.481963    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvfck\" (UniqueName: \"kubernetes.io/projected/5ec89a7d-eb27-48de-a793-86a7b0d5389b-kube-api-access-wvfck\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.482015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482001    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b26525-0a60-471e-8fe4-2f5a9696b700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.482229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482079    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d67356-8625-4b93-88fe-b33f21dc9b8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.482229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482121    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6z8j\" (UniqueName: \"kubernetes.io/projected/81d67356-8625-4b93-88fe-b33f21dc9b8e-kube-api-access-r6z8j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.482229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482171    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89a7d-eb27-48de-a793-86a7b0d5389b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.482397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482253    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njxl4\" (UniqueName: \"kubernetes.io/projected/32860ef1-4fcb-45f2-8297-f3330e738442-kube-api-access-njxl4\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.482397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482289    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f8wd\" (UniqueName: \"kubernetes.io/projected/696f6bbf-ec44-4546-8d71-29bd05cf026e-kube-api-access-8f8wd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.482397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482324    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.483551 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.482914    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714a3ae-1af2-43fd-824c-e6536556a50f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.483551 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.483212    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.483728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.483630    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32860ef1-4fcb-45f2-8297-f3330e738442-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.483999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.483974    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89a7d-eb27-48de-a793-86a7b0d5389b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.484976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.484907    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b26525-0a60-471e-8fe4-2f5a9696b700-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.487017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.486997    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"]
Oct 30 13:45:00.487320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.487302    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.492451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.492152    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"]
Oct 30 13:45:00.507366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.505905    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"]
Oct 30 13:45:00.509714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.508737    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrn6d\" (UniqueName: \"kubernetes.io/projected/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-kube-api-access-lrn6d\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.514512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.513145    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"]
Oct 30 13:45:00.514512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.513422    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tw6z8\" (UniqueName: \"kubernetes.io/projected/5714a3ae-1af2-43fd-824c-e6536556a50f-kube-api-access-tw6z8\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.516589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.516171    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvfck\" (UniqueName: \"kubernetes.io/projected/5ec89a7d-eb27-48de-a793-86a7b0d5389b-kube-api-access-wvfck\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.516589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.516220    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njxl4\" (UniqueName: \"kubernetes.io/projected/32860ef1-4fcb-45f2-8297-f3330e738442-kube-api-access-njxl4\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.517527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.517391    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:00.518285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.518260    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgf2z\" (UniqueName: \"kubernetes.io/projected/73b26525-0a60-471e-8fe4-2f5a9696b700-kube-api-access-zgf2z\") pod \"maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.519307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.519290    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"]
Oct 30 13:45:00.519405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.519370    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.524385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.524364    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:00.528861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.528842    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"]
Oct 30 13:45:00.528935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.528885    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"]
Oct 30 13:45:00.529069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.528975    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.532837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.532813    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"]
Oct 30 13:45:00.540144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.539969    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:00.542277 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.542260    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:00.588619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.588572    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb4hk\" (UniqueName: \"kubernetes.io/projected/80678d19-5845-487b-9329-577e2cf9738e-kube-api-access-tb4hk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.588803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.588670    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46831069-b252-4c2d-87d4-834789ce4d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.588803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.588702    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppzhc\" (UniqueName: \"kubernetes.io/projected/7253bff1-99f9-4619-9239-3a76834a05dc-kube-api-access-ppzhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.588803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.588758    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75451d0a-8ddb-411c-98c7-b4fa2101d048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.589036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.588871    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4206617-2601-491e-a274-cb89b62cc103-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:00.589036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.588964    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d67356-8625-4b93-88fe-b33f21dc9b8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.589036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.588998    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6z8j\" (UniqueName: \"kubernetes.io/projected/81d67356-8625-4b93-88fe-b33f21dc9b8e-kube-api-access-r6z8j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.589199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589075    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f8wd\" (UniqueName: \"kubernetes.io/projected/696f6bbf-ec44-4546-8d71-29bd05cf026e-kube-api-access-8f8wd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.589199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589100    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.589199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589137    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd06531d-fb03-48a0-864d-b3e33da93784-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-mt5wq\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.589199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589171    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld9wc\" (UniqueName: \"kubernetes.io/projected/392207f2-fca4-4e13-80b4-218ff561c46f-kube-api-access-ld9wc\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.589407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589203    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn8hk\" (UniqueName: \"kubernetes.io/projected/75451d0a-8ddb-411c-98c7-b4fa2101d048-kube-api-access-sn8hk\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.589407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589229    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696f6bbf-ec44-4546-8d71-29bd05cf026e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.589407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589291    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcp6p\" (UniqueName: \"kubernetes.io/projected/46831069-b252-4c2d-87d4-834789ce4d6e-kube-api-access-rcp6p\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.589407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589331    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt8tf\" (UniqueName: \"kubernetes.io/projected/ba738823-8bdf-4597-8eed-7e3a3fb027f2-kube-api-access-rt8tf\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:00.589407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589381    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80678d19-5845-487b-9329-577e2cf9738e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.589720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589412    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5gbv\" (UniqueName: \"kubernetes.io/projected/b4206617-2601-491e-a274-cb89b62cc103-kube-api-access-q5gbv\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:00.589720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589456    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba738823-8bdf-4597-8eed-7e3a3fb027f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:00.589720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589487    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7253bff1-99f9-4619-9239-3a76834a05dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.589720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589521    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bsrpp\" (UniqueName: \"kubernetes.io/projected/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-kube-api-access-bsrpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.589720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589550    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:00.589720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589594    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/392207f2-fca4-4e13-80b4-218ff561c46f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.589720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589681    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77k4h\" (UniqueName: \"kubernetes.io/projected/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-kube-api-access-77k4h\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:00.590397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.589723    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsxtk\" (UniqueName: \"kubernetes.io/projected/fd06531d-fb03-48a0-864d-b3e33da93784-kube-api-access-lsxtk\") pod \"maas-api-key-cleanup-29889465-mt5wq\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.590553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.590530    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80678d19-5845-487b-9329-577e2cf9738e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.590646 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.590605    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d67356-8625-4b93-88fe-b33f21dc9b8e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.594590 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.594215    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696f6bbf-ec44-4546-8d71-29bd05cf026e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.595666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.594759    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.653354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.653320    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6z8j\" (UniqueName: \"kubernetes.io/projected/81d67356-8625-4b93-88fe-b33f21dc9b8e-kube-api-access-r6z8j\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.653513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.653320    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb4hk\" (UniqueName: \"kubernetes.io/projected/80678d19-5845-487b-9329-577e2cf9738e-kube-api-access-tb4hk\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.653513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.653320    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bsrpp\" (UniqueName: \"kubernetes.io/projected/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-kube-api-access-bsrpp\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.657055 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.657023    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f8wd\" (UniqueName: \"kubernetes.io/projected/696f6bbf-ec44-4546-8d71-29bd05cf026e-kube-api-access-8f8wd\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.662081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.661854    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:00.670967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.670013    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:00.686487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.686162    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.690586    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt8tf\" (UniqueName: \"kubernetes.io/projected/ba738823-8bdf-4597-8eed-7e3a3fb027f2-kube-api-access-rt8tf\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.690660    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5gbv\" (UniqueName: \"kubernetes.io/projected/b4206617-2601-491e-a274-cb89b62cc103-kube-api-access-q5gbv\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.690704    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba738823-8bdf-4597-8eed-7e3a3fb027f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.690734    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7253bff1-99f9-4619-9239-3a76834a05dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.690769    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691026    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/392207f2-fca4-4e13-80b4-218ff561c46f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691071    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77k4h\" (UniqueName: \"kubernetes.io/projected/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-kube-api-access-77k4h\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691113    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsxtk\" (UniqueName: \"kubernetes.io/projected/fd06531d-fb03-48a0-864d-b3e33da93784-kube-api-access-lsxtk\") pod \"maas-api-key-cleanup-29889465-mt5wq\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691183    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46831069-b252-4c2d-87d4-834789ce4d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691209    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppzhc\" (UniqueName: \"kubernetes.io/projected/7253bff1-99f9-4619-9239-3a76834a05dc-kube-api-access-ppzhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691274    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75451d0a-8ddb-411c-98c7-b4fa2101d048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691313    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4206617-2601-491e-a274-cb89b62cc103-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691461    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd06531d-fb03-48a0-864d-b3e33da93784-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-mt5wq\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691499    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld9wc\" (UniqueName: \"kubernetes.io/projected/392207f2-fca4-4e13-80b4-218ff561c46f-kube-api-access-ld9wc\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691522    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691523    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba738823-8bdf-4597-8eed-7e3a3fb027f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691527    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn8hk\" (UniqueName: \"kubernetes.io/projected/75451d0a-8ddb-411c-98c7-b4fa2101d048-kube-api-access-sn8hk\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691562    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7253bff1-99f9-4619-9239-3a76834a05dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.691659    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcp6p\" (UniqueName: \"kubernetes.io/projected/46831069-b252-4c2d-87d4-834789ce4d6e-kube-api-access-rcp6p\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.692265    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46831069-b252-4c2d-87d4-834789ce4d6e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.692385    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75451d0a-8ddb-411c-98c7-b4fa2101d048-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.692864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.692443    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd06531d-fb03-48a0-864d-b3e33da93784-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889465-mt5wq\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.693659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.692912    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4206617-2601-491e-a274-cb89b62cc103-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:00.693659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.693084    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/392207f2-fca4-4e13-80b4-218ff561c46f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.723714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.720933    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcp6p\" (UniqueName: \"kubernetes.io/projected/46831069-b252-4c2d-87d4-834789ce4d6e-kube-api-access-rcp6p\") pod \"maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.756466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.755780    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld9wc\" (UniqueName: \"kubernetes.io/projected/392207f2-fca4-4e13-80b4-218ff561c46f-kube-api-access-ld9wc\") pod \"maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.756466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.756326    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77k4h\" (UniqueName: \"kubernetes.io/projected/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-kube-api-access-77k4h\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:00.756466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.756418    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn8hk\" (UniqueName: \"kubernetes.io/projected/75451d0a-8ddb-411c-98c7-b4fa2101d048-kube-api-access-sn8hk\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.758353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.758289    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsxtk\" (UniqueName: \"kubernetes.io/projected/fd06531d-fb03-48a0-864d-b3e33da93784-kube-api-access-lsxtk\") pod \"maas-api-key-cleanup-29889465-mt5wq\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.758601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.758578    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt8tf\" (UniqueName: \"kubernetes.io/projected/ba738823-8bdf-4597-8eed-7e3a3fb027f2-kube-api-access-rt8tf\") pod \"maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:00.762509 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.762301    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppzhc\" (UniqueName: \"kubernetes.io/projected/7253bff1-99f9-4619-9239-3a76834a05dc-kube-api-access-ppzhc\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.763093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.763066    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5gbv\" (UniqueName: \"kubernetes.io/projected/b4206617-2601-491e-a274-cb89b62cc103-kube-api-access-q5gbv\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:00.801326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.801289    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:00.808328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.807492    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:00.812031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.812005    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:00.866999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.866953    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"]
Oct 30 13:45:00.868240 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:45:00.868209    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ec89a7d_eb27_48de_a793_86a7b0d5389b.slice/crio-77f163fe2a669deeba08275a71498fbb7566fc475a1b653161b431299533b4a7 WatchSource:0}: Error finding container 77f163fe2a669deeba08275a71498fbb7566fc475a1b653161b431299533b4a7: Status 404 returned error can't find the container with id 77f163fe2a669deeba08275a71498fbb7566fc475a1b653161b431299533b4a7
Oct 30 13:45:00.922138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.922093    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"]
Oct 30 13:45:00.933989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.933903    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:00.936934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.936736    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:00.944688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.943612    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"]
Oct 30 13:45:00.944688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.944025    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:00.948176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.945145    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:00.965862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.965130    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"]
Oct 30 13:45:00.995461 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:00.995244    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:00.996071 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:45:00.996031    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28a7ba16_b61c_4139_b1a1_fa2de8b12fa4.slice/crio-c868f9a5b12bded6ccdbb3a5fed3381d553c359e30ed3c72c8547cf993d4062b WatchSource:0}: Error finding container c868f9a5b12bded6ccdbb3a5fed3381d553c359e30ed3c72c8547cf993d4062b: Status 404 returned error can't find the container with id c868f9a5b12bded6ccdbb3a5fed3381d553c359e30ed3c72c8547cf993d4062b
Oct 30 13:45:01.013468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.013307    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:01.019414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.019042    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:01.044301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.042955    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:01.067952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.067779    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"]
Oct 30 13:45:01.096719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.096690    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"]
Oct 30 13:45:01.101350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101174    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101252    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101291    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101335    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101403    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bc2ea8-1f71-4718-b322-80dcc69af66a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101523    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101550    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101579    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcbl6\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-kube-api-access-wcbl6\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.101724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.101693    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.156361 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:45:01.156132    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5714a3ae_1af2_43fd_824c_e6536556a50f.slice/crio-3c23d390516d2f3d7fe4d2f11764565ac1df22f363b4c7c650d940edddac8a41 WatchSource:0}: Error finding container 3c23d390516d2f3d7fe4d2f11764565ac1df22f363b4c7c650d940edddac8a41: Status 404 returned error can't find the container with id 3c23d390516d2f3d7fe4d2f11764565ac1df22f363b4c7c650d940edddac8a41
Oct 30 13:45:01.205803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.204232    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.205803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.204301    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.205803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.204373    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.205803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.204404    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bc2ea8-1f71-4718-b322-80dcc69af66a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.205803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.205463    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.205803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.205479    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"]
Oct 30 13:45:01.205803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.205746    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.207660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.206489    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bc2ea8-1f71-4718-b322-80dcc69af66a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.208730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.207830    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.208730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.207882    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.208730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.207930    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcbl6\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-kube-api-access-wcbl6\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.208730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.208063    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.208730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.208145    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.208730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.208545    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.210236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.209349    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.213535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.213509    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.220067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.220025    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.238440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.238361    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"]
Oct 30 13:45:01.246029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.245655    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"]
Oct 30 13:45:01.257933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.257875    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.266261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.264140    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcbl6\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-kube-api-access-wcbl6\") pod \"maas-default-gateway-openshift-default-67bf986f99-pbpdb\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.319811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.319733    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" event={"ID":"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4","Type":"ContainerStarted","Data":"c868f9a5b12bded6ccdbb3a5fed3381d553c359e30ed3c72c8547cf993d4062b"}
Oct 30 13:45:01.323819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.323782    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" event={"ID":"5714a3ae-1af2-43fd-824c-e6536556a50f","Type":"ContainerStarted","Data":"3c23d390516d2f3d7fe4d2f11764565ac1df22f363b4c7c650d940edddac8a41"}
Oct 30 13:45:01.344174 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.344099    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" event={"ID":"5ec89a7d-eb27-48de-a793-86a7b0d5389b","Type":"ContainerStarted","Data":"77f163fe2a669deeba08275a71498fbb7566fc475a1b653161b431299533b4a7"}
Oct 30 13:45:01.347519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.347486    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" event={"ID":"73b26525-0a60-471e-8fe4-2f5a9696b700","Type":"ContainerStarted","Data":"0dc642e108e6ea764b53e50cf644847adea50d043cf5a3e8044747fcb49ee965"}
Oct 30 13:45:01.350368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.350324    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:01.379018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.368568    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"]
Oct 30 13:45:01.384041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.384013    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"]
Oct 30 13:45:01.416117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.416089    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"]
Oct 30 13:45:01.545232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.545201    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"]
Oct 30 13:45:01.558072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.558040    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"]
Oct 30 13:45:01.587188 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:45:01.585554    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7253bff1_99f9_4619_9239_3a76834a05dc.slice/crio-4ed0b0b974247a7acd6daa3f08f59fd7a05b07a0d39fddccbb30a532e37e89b7 WatchSource:0}: Error finding container 4ed0b0b974247a7acd6daa3f08f59fd7a05b07a0d39fddccbb30a532e37e89b7: Status 404 returned error can't find the container with id 4ed0b0b974247a7acd6daa3f08f59fd7a05b07a0d39fddccbb30a532e37e89b7
Oct 30 13:45:01.619664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.614333    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"]
Oct 30 13:45:01.635919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.635886    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"]
Oct 30 13:45:01.675676 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.675649    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"]
Oct 30 13:45:01.686371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.686006    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"]
Oct 30 13:45:01.714576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.714455    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"]
Oct 30 13:45:01.949013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:01.948979    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"]
Oct 30 13:45:02.177233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.177197    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:45:02.274439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.273914    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:45:02.274439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.274019    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:45:02.274439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.274063    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:45:02.362216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.362183    4291 generic.go:356] "Generic (PLEG): container finished" podID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerID="48e5d4f3c3a342a542b5da3700933754c571b70086ae9efe066ec602331b7c05" exitCode=60
Oct 30 13:45:02.362343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.362291    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" event={"ID":"5ec89a7d-eb27-48de-a793-86a7b0d5389b","Type":"ContainerDied","Data":"48e5d4f3c3a342a542b5da3700933754c571b70086ae9efe066ec602331b7c05"}
Oct 30 13:45:02.363073 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.363052    4291 scope.go:122] "RemoveContainer" containerID="48e5d4f3c3a342a542b5da3700933754c571b70086ae9efe066ec602331b7c05"
Oct 30 13:45:02.365226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.365200    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerStarted","Data":"b64773558eb77896822e3584fa5f61259b1cefee228143f2d7e2feef916cdc40"}
Oct 30 13:45:02.367022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.366992    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerStarted","Data":"4ed0b0b974247a7acd6daa3f08f59fd7a05b07a0d39fddccbb30a532e37e89b7"}
Oct 30 13:45:02.372333 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.372301    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" event={"ID":"73b26525-0a60-471e-8fe4-2f5a9696b700","Type":"ContainerStarted","Data":"07db47080aa63de73eca9ce6e701390986b5f5d587cccc39ef952e06c5970128"}
Oct 30 13:45:02.389965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.389935    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" event={"ID":"392207f2-fca4-4e13-80b4-218ff561c46f","Type":"ContainerStarted","Data":"b27e72d6e2fed62a2a3d78718227bf113afbfa29eedf6a43874c1d8da0a2fd3b"}
Oct 30 13:45:02.392387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.392360    4291 generic.go:356] "Generic (PLEG): container finished" podID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerID="7011aa0508b3897b5f3cf512052fb1e010a301f4038ca8e052011b4729f66900" exitCode=60
Oct 30 13:45:02.392505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.392438    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" event={"ID":"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4","Type":"ContainerDied","Data":"7011aa0508b3897b5f3cf512052fb1e010a301f4038ca8e052011b4729f66900"}
Oct 30 13:45:02.392742 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.392724    4291 scope.go:122] "RemoveContainer" containerID="7011aa0508b3897b5f3cf512052fb1e010a301f4038ca8e052011b4729f66900"
Oct 30 13:45:02.394014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.393989    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" event={"ID":"32860ef1-4fcb-45f2-8297-f3330e738442","Type":"ContainerStarted","Data":"9992d7d6cd09087449a29600c382e8d5c603caddce15a457f184f54c175368a1"}
Oct 30 13:45:02.395388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.395358    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" event={"ID":"46831069-b252-4c2d-87d4-834789ce4d6e","Type":"ContainerStarted","Data":"0eda99b3eb877f61adc7db0bd89f03a71ec01ad8da9d6b0b7728921e7fceac42"}
Oct 30 13:45:02.396597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.396571    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerStarted","Data":"2506af3636247ca17b22b2f092723082e4e6c7d51fe2f3256a8609e30483d9b1"}
Oct 30 13:45:02.400482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.400464    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" event={"ID":"b4206617-2601-491e-a274-cb89b62cc103","Type":"ContainerStarted","Data":"b28934a959ebd5013b86ee703ccf37ddf5f406c5e5a0a29be25ef0d7d70a285c"}
Oct 30 13:45:02.402320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.402286    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" event={"ID":"81d67356-8625-4b93-88fe-b33f21dc9b8e","Type":"ContainerStarted","Data":"0b135215d671e1e231395f17c01a21f88991a6c016fd66584e3e5eebad029890"}
Oct 30 13:45:02.403828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.403776    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerStarted","Data":"d0b22c0a6dd4a56eb1c0eaa05c5ef5fb14907780ff789e18fc4697039afcbc4c"}
Oct 30 13:45:02.406023 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.405406    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" event={"ID":"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b","Type":"ContainerStarted","Data":"09fbcf628a9a55f77eb78d188e1e89d853a08bade7c54905106098a1632ff175"}
Oct 30 13:45:02.412273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.412245    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" event={"ID":"5714a3ae-1af2-43fd-824c-e6536556a50f","Type":"ContainerStarted","Data":"eed00361a53aa37fb919b67919bf4e7c2d691009044264978f14f1b932c8985d"}
Oct 30 13:45:02.414627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.414598    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" event={"ID":"fd06531d-fb03-48a0-864d-b3e33da93784","Type":"ContainerStarted","Data":"d68040ae3106f8a320955b286e4dd358b4e74ab14cd350f8a304972f660570bc"}
Oct 30 13:45:02.416851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.416796    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" event={"ID":"696f6bbf-ec44-4546-8d71-29bd05cf026e","Type":"ContainerStarted","Data":"d8d8d8d325211367ec62d3fbacf25cc0d4f60ca2e80fdff40bee84f93cbaad07"}
Oct 30 13:45:02.418756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.418731    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerStarted","Data":"e109526449e6586066a75e3364bdc1e10c7ae46edb2a38591386cb4fbba9d2d5"}
Oct 30 13:45:02.418857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.418778    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerStarted","Data":"74401d8ad9997aa06fb40ef5cb6abde513ded966dbdb112bf99efa4ddf94a447"}
Oct 30 13:45:02.421037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.421003    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" event={"ID":"83bc2ea8-1f71-4718-b322-80dcc69af66a","Type":"ContainerStarted","Data":"0b69509512a849652c8175f4ccae1b5bd9adb05f629b1316887018525da1aa5c"}
Oct 30 13:45:02.501969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.501905    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" podStartSLOduration=2.501891005 podStartE2EDuration="2.501891005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.496871868 +0000 UTC m=+2512.757110102" watchObservedRunningTime="2026-10-30 13:45:02.501891005 +0000 UTC m=+2512.762129238"
Oct 30 13:45:02.544497 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.544385    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" podStartSLOduration=2.544368493 podStartE2EDuration="2.544368493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.542753528 +0000 UTC m=+2512.802991762" watchObservedRunningTime="2026-10-30 13:45:02.544368493 +0000 UTC m=+2512.804606724"
Oct 30 13:45:02.587717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.587663    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" podStartSLOduration=2.5876477749999998 podStartE2EDuration="2.587647775s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.586358377 +0000 UTC m=+2512.846596634" watchObservedRunningTime="2026-10-30 13:45:02.587647775 +0000 UTC m=+2512.847886014"
Oct 30 13:45:02.652661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:02.652229    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" podStartSLOduration=2.652208205 podStartE2EDuration="2.652208205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.645122225 +0000 UTC m=+2512.905360457" watchObservedRunningTime="2026-10-30 13:45:02.652208205 +0000 UTC m=+2512.912446438"
Oct 30 13:45:03.444076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.444034    4291 generic.go:356] "Generic (PLEG): container finished" podID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerID="77d5c2f037d2951eb1c50f9b2334629ced06dbfdd4f519c92f90d4b61baf226b" exitCode=60
Oct 30 13:45:03.444797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.444459    4291 scope.go:122] "RemoveContainer" containerID="77d5c2f037d2951eb1c50f9b2334629ced06dbfdd4f519c92f90d4b61baf226b"
Oct 30 13:45:03.445086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.444524    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" event={"ID":"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b","Type":"ContainerDied","Data":"77d5c2f037d2951eb1c50f9b2334629ced06dbfdd4f519c92f90d4b61baf226b"}
Oct 30 13:45:03.472655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.472308    4291 generic.go:356] "Generic (PLEG): container finished" podID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerID="eed00361a53aa37fb919b67919bf4e7c2d691009044264978f14f1b932c8985d" exitCode=60
Oct 30 13:45:03.472655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.472417    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" event={"ID":"5714a3ae-1af2-43fd-824c-e6536556a50f","Type":"ContainerDied","Data":"eed00361a53aa37fb919b67919bf4e7c2d691009044264978f14f1b932c8985d"}
Oct 30 13:45:03.472898 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.472841    4291 scope.go:122] "RemoveContainer" containerID="eed00361a53aa37fb919b67919bf4e7c2d691009044264978f14f1b932c8985d"
Oct 30 13:45:03.474384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.474348    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd06531d-fb03-48a0-864d-b3e33da93784" containerID="c97361b0ffde98748dcb8624dce4605da0b97da2c7fd71917b08256cf33c986f" exitCode=60
Oct 30 13:45:03.475697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.474798    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" event={"ID":"fd06531d-fb03-48a0-864d-b3e33da93784","Type":"ContainerDied","Data":"c97361b0ffde98748dcb8624dce4605da0b97da2c7fd71917b08256cf33c986f"}
Oct 30 13:45:03.475697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.475082    4291 scope.go:122] "RemoveContainer" containerID="c97361b0ffde98748dcb8624dce4605da0b97da2c7fd71917b08256cf33c986f"
Oct 30 13:45:03.477573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.476865    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" event={"ID":"696f6bbf-ec44-4546-8d71-29bd05cf026e","Type":"ContainerStarted","Data":"c4179efa965cf9062a4a2fec0c57dd6c8d7ebc8981c464738d4a06cf905690be"}
Oct 30 13:45:03.478204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.477979    4291 scope.go:122] "RemoveContainer" containerID="c4179efa965cf9062a4a2fec0c57dd6c8d7ebc8981c464738d4a06cf905690be"
Oct 30 13:45:03.480905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.480885    4291 generic.go:356] "Generic (PLEG): container finished" podID="80678d19-5845-487b-9329-577e2cf9738e" containerID="e109526449e6586066a75e3364bdc1e10c7ae46edb2a38591386cb4fbba9d2d5" exitCode=60
Oct 30 13:45:03.483044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.481187    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerDied","Data":"e109526449e6586066a75e3364bdc1e10c7ae46edb2a38591386cb4fbba9d2d5"}
Oct 30 13:45:03.483044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.482998    4291 scope.go:122] "RemoveContainer" containerID="e109526449e6586066a75e3364bdc1e10c7ae46edb2a38591386cb4fbba9d2d5"
Oct 30 13:45:03.485081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.485028    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" event={"ID":"83bc2ea8-1f71-4718-b322-80dcc69af66a","Type":"ContainerStarted","Data":"1d1b8a2f0c4579038804a015157e4b0afbcb3126fd30009d5f6601787405bc09"}
Oct 30 13:45:03.489410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.488816    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" event={"ID":"5ec89a7d-eb27-48de-a793-86a7b0d5389b","Type":"ContainerStarted","Data":"c3baf89fc92dcd79252a477d3210cf31539b3b49da39375dbc330c1ea1100170"}
Oct 30 13:45:03.491860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.491604    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerStarted","Data":"8f48a7494083c80eed6d4dd58344d3f6d0fb84407a08eaefdab8676aac029273"}
Oct 30 13:45:03.494355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.494330    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerStarted","Data":"055cc1b3e2ba9d8010ec736fd8b0c8cf6573ca3dcf49bf44002274f26311a3ad"}
Oct 30 13:45:03.496942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.496538    4291 generic.go:356] "Generic (PLEG): container finished" podID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerID="07db47080aa63de73eca9ce6e701390986b5f5d587cccc39ef952e06c5970128" exitCode=60
Oct 30 13:45:03.496942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.496616    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" event={"ID":"73b26525-0a60-471e-8fe4-2f5a9696b700","Type":"ContainerDied","Data":"07db47080aa63de73eca9ce6e701390986b5f5d587cccc39ef952e06c5970128"}
Oct 30 13:45:03.496942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.496833    4291 scope.go:122] "RemoveContainer" containerID="07db47080aa63de73eca9ce6e701390986b5f5d587cccc39ef952e06c5970128"
Oct 30 13:45:03.504693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.498815    4291 generic.go:356] "Generic (PLEG): container finished" podID="392207f2-fca4-4e13-80b4-218ff561c46f" containerID="262f0ca81d29d52c307b4d527b5e8688f27adab2e505300b65a22b9972ebcfee" exitCode=60
Oct 30 13:45:03.504693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.498899    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" event={"ID":"392207f2-fca4-4e13-80b4-218ff561c46f","Type":"ContainerDied","Data":"262f0ca81d29d52c307b4d527b5e8688f27adab2e505300b65a22b9972ebcfee"}
Oct 30 13:45:03.504693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.499167    4291 scope.go:122] "RemoveContainer" containerID="262f0ca81d29d52c307b4d527b5e8688f27adab2e505300b65a22b9972ebcfee"
Oct 30 13:45:03.504693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.501932    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" event={"ID":"32860ef1-4fcb-45f2-8297-f3330e738442","Type":"ContainerStarted","Data":"7b501803741619e343054c7f8b7435b4b0ba3feb7eb8e6a5496f65bc56d11626"}
Oct 30 13:45:03.504693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.503184    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" podStartSLOduration=3.503168342 podStartE2EDuration="3.503168342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:02.695871154 +0000 UTC m=+2512.956109414" watchObservedRunningTime="2026-10-30 13:45:03.503168342 +0000 UTC m=+2513.763406577"
Oct 30 13:45:03.505718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.505332    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" event={"ID":"46831069-b252-4c2d-87d4-834789ce4d6e","Type":"ContainerStarted","Data":"e21fc35d31ce2725ae24b64fff7e5bf0d4a484a2f5b6d0d0bfcfeb52c3e5e681"}
Oct 30 13:45:03.507312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.507213    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerStarted","Data":"998d7939fa237a077fcec64673e75b65d915056e52cd9d42d1c67c5bc05a88aa"}
Oct 30 13:45:03.509420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.509401    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" event={"ID":"b4206617-2601-491e-a274-cb89b62cc103","Type":"ContainerStarted","Data":"60c385250faee45c69234475c2c0b5a7a32cbfa68a12a2b88ecdac60ce0bf287"}
Oct 30 13:45:03.523934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.523895    4291 generic.go:356] "Generic (PLEG): container finished" podID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerID="015ed6f4c4b91fb7d5050ed189edfadb317b298493107a0fa270a225fa84b826" exitCode=60
Oct 30 13:45:03.524056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.524030    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" event={"ID":"81d67356-8625-4b93-88fe-b33f21dc9b8e","Type":"ContainerDied","Data":"015ed6f4c4b91fb7d5050ed189edfadb317b298493107a0fa270a225fa84b826"}
Oct 30 13:45:03.524293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.524275    4291 scope.go:122] "RemoveContainer" containerID="015ed6f4c4b91fb7d5050ed189edfadb317b298493107a0fa270a225fa84b826"
Oct 30 13:45:03.527941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.527914    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerStarted","Data":"380f3ce46f1c1b30bf34d272bed9db6894144d217a26892fe8fd1006cf31db5e"}
Oct 30 13:45:03.616888 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.616832    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podStartSLOduration=3.6168161789999997 podStartE2EDuration="3.616816179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.613597685 +0000 UTC m=+2513.873835922" watchObservedRunningTime="2026-10-30 13:45:03.616816179 +0000 UTC m=+2513.877054477"
Oct 30 13:45:03.829721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:03.829660    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" podStartSLOduration=3.829627523 podStartE2EDuration="3.829627523s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:03.826252629 +0000 UTC m=+2514.086490863" watchObservedRunningTime="2026-10-30 13:45:03.829627523 +0000 UTC m=+2514.089865757"
Oct 30 13:45:04.086627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.086536    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" podStartSLOduration=4.086512383 podStartE2EDuration="4.086512383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.071205722 +0000 UTC m=+2514.331443957" watchObservedRunningTime="2026-10-30 13:45:04.086512383 +0000 UTC m=+2514.346750620"
Oct 30 13:45:04.133331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.133278    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" podStartSLOduration=4.133262387 podStartE2EDuration="4.133262387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.128803918 +0000 UTC m=+2514.389042152" watchObservedRunningTime="2026-10-30 13:45:04.133262387 +0000 UTC m=+2514.393500633"
Oct 30 13:45:04.318456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.318406    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" podStartSLOduration=4.318391322 podStartE2EDuration="4.318391322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.317075362 +0000 UTC m=+2514.577313591" watchObservedRunningTime="2026-10-30 13:45:04.318391322 +0000 UTC m=+2514.578629559"
Oct 30 13:45:04.359075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.359039    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:04.373662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.371974    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:04.373662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.372047    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:04.383367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.383310    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" podStartSLOduration=4.383294767 podStartE2EDuration="4.383294767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.380546775 +0000 UTC m=+2514.640785012" watchObservedRunningTime="2026-10-30 13:45:04.383294767 +0000 UTC m=+2514.643533001"
Oct 30 13:45:04.442172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.442055    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" podStartSLOduration=4.44204035 podStartE2EDuration="4.44204035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.438549407 +0000 UTC m=+2514.698787640" watchObservedRunningTime="2026-10-30 13:45:04.44204035 +0000 UTC m=+2514.702278583"
Oct 30 13:45:04.538671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.538608    4291 generic.go:356] "Generic (PLEG): container finished" podID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerID="380f3ce46f1c1b30bf34d272bed9db6894144d217a26892fe8fd1006cf31db5e" exitCode=60
Oct 30 13:45:04.538671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.538664    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerDied","Data":"380f3ce46f1c1b30bf34d272bed9db6894144d217a26892fe8fd1006cf31db5e"}
Oct 30 13:45:04.539216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.539071    4291 scope.go:122] "RemoveContainer" containerID="380f3ce46f1c1b30bf34d272bed9db6894144d217a26892fe8fd1006cf31db5e"
Oct 30 13:45:04.541228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.541202    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" event={"ID":"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b","Type":"ContainerStarted","Data":"43ad622d0989d24612e1d19e4268a260ab5a7d90abd36670f2bd98707b14d74c"}
Oct 30 13:45:04.543127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.543105    4291 generic.go:356] "Generic (PLEG): container finished" podID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerID="c4179efa965cf9062a4a2fec0c57dd6c8d7ebc8981c464738d4a06cf905690be" exitCode=60
Oct 30 13:45:04.543231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.543200    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" event={"ID":"696f6bbf-ec44-4546-8d71-29bd05cf026e","Type":"ContainerDied","Data":"c4179efa965cf9062a4a2fec0c57dd6c8d7ebc8981c464738d4a06cf905690be"}
Oct 30 13:45:04.545278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.545251    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerStarted","Data":"81353d5f80abbbaf30cb3594e72d600e81db86790df954bb4a91044a36becc58"}
Oct 30 13:45:04.547293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.547269    4291 generic.go:356] "Generic (PLEG): container finished" podID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerID="c3baf89fc92dcd79252a477d3210cf31539b3b49da39375dbc330c1ea1100170" exitCode=60
Oct 30 13:45:04.547389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.547341    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" event={"ID":"5ec89a7d-eb27-48de-a793-86a7b0d5389b","Type":"ContainerDied","Data":"c3baf89fc92dcd79252a477d3210cf31539b3b49da39375dbc330c1ea1100170"}
Oct 30 13:45:04.547453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.547388    4291 scope.go:122] "RemoveContainer" containerID="48e5d4f3c3a342a542b5da3700933754c571b70086ae9efe066ec602331b7c05"
Oct 30 13:45:04.547857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.547533    4291 scope.go:122] "RemoveContainer" containerID="c3baf89fc92dcd79252a477d3210cf31539b3b49da39375dbc330c1ea1100170"
Oct 30 13:45:04.547857 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:04.547816    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz_odh-ai-gateway-infra(5ec89a7d-eb27-48de-a793-86a7b0d5389b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b"
Oct 30 13:45:04.549141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.549117    4291 generic.go:356] "Generic (PLEG): container finished" podID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerID="8f48a7494083c80eed6d4dd58344d3f6d0fb84407a08eaefdab8676aac029273" exitCode=60
Oct 30 13:45:04.549227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.549189    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerDied","Data":"8f48a7494083c80eed6d4dd58344d3f6d0fb84407a08eaefdab8676aac029273"}
Oct 30 13:45:04.549388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.549373    4291 scope.go:122] "RemoveContainer" containerID="8f48a7494083c80eed6d4dd58344d3f6d0fb84407a08eaefdab8676aac029273"
Oct 30 13:45:04.584984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.584940    4291 generic.go:356] "Generic (PLEG): container finished" podID="7253bff1-99f9-4619-9239-3a76834a05dc" containerID="055cc1b3e2ba9d8010ec736fd8b0c8cf6573ca3dcf49bf44002274f26311a3ad" exitCode=60
Oct 30 13:45:04.585187 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.585127    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerDied","Data":"055cc1b3e2ba9d8010ec736fd8b0c8cf6573ca3dcf49bf44002274f26311a3ad"}
Oct 30 13:45:04.585845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.585800    4291 scope.go:122] "RemoveContainer" containerID="055cc1b3e2ba9d8010ec736fd8b0c8cf6573ca3dcf49bf44002274f26311a3ad"
Oct 30 13:45:04.611568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.611531    4291 generic.go:356] "Generic (PLEG): container finished" podID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerID="ad148db728c74f34c8468c6aeae6384b57ee490c3fbcc3383db74938b18fda11" exitCode=60
Oct 30 13:45:04.612057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.612027    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" event={"ID":"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4","Type":"ContainerDied","Data":"ad148db728c74f34c8468c6aeae6384b57ee490c3fbcc3383db74938b18fda11"}
Oct 30 13:45:04.612589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.612565    4291 scope.go:122] "RemoveContainer" containerID="ad148db728c74f34c8468c6aeae6384b57ee490c3fbcc3383db74938b18fda11"
Oct 30 13:45:04.613954 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:04.612992    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz_odh-ai-gateway-infra(28a7ba16-b61c-4139-b1a1-fa2de8b12fa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4"
Oct 30 13:45:04.614577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.614520    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" podStartSLOduration=4.6145014589999995 podStartE2EDuration="4.614501459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.605982948 +0000 UTC m=+2514.866221184" watchObservedRunningTime="2026-10-30 13:45:04.614501459 +0000 UTC m=+2514.874739695"
Oct 30 13:45:04.622339 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.622309    4291 scope.go:122] "RemoveContainer" containerID="7011aa0508b3897b5f3cf512052fb1e010a301f4038ca8e052011b4729f66900"
Oct 30 13:45:04.625034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.625005    4291 generic.go:356] "Generic (PLEG): container finished" podID="32860ef1-4fcb-45f2-8297-f3330e738442" containerID="7b501803741619e343054c7f8b7435b4b0ba3feb7eb8e6a5496f65bc56d11626" exitCode=60
Oct 30 13:45:04.625133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.625117    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" event={"ID":"32860ef1-4fcb-45f2-8297-f3330e738442","Type":"ContainerDied","Data":"7b501803741619e343054c7f8b7435b4b0ba3feb7eb8e6a5496f65bc56d11626"}
Oct 30 13:45:04.625511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.625492    4291 scope.go:122] "RemoveContainer" containerID="7b501803741619e343054c7f8b7435b4b0ba3feb7eb8e6a5496f65bc56d11626"
Oct 30 13:45:04.629763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.629018    4291 generic.go:356] "Generic (PLEG): container finished" podID="46831069-b252-4c2d-87d4-834789ce4d6e" containerID="e21fc35d31ce2725ae24b64fff7e5bf0d4a484a2f5b6d0d0bfcfeb52c3e5e681" exitCode=60
Oct 30 13:45:04.629763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.629108    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" event={"ID":"46831069-b252-4c2d-87d4-834789ce4d6e","Type":"ContainerDied","Data":"e21fc35d31ce2725ae24b64fff7e5bf0d4a484a2f5b6d0d0bfcfeb52c3e5e681"}
Oct 30 13:45:04.629763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.629473    4291 scope.go:122] "RemoveContainer" containerID="e21fc35d31ce2725ae24b64fff7e5bf0d4a484a2f5b6d0d0bfcfeb52c3e5e681"
Oct 30 13:45:04.643394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.643361    4291 generic.go:356] "Generic (PLEG): container finished" podID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerID="998d7939fa237a077fcec64673e75b65d915056e52cd9d42d1c67c5bc05a88aa" exitCode=60
Oct 30 13:45:04.644001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.643901    4291 scope.go:122] "RemoveContainer" containerID="998d7939fa237a077fcec64673e75b65d915056e52cd9d42d1c67c5bc05a88aa"
Oct 30 13:45:04.644394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.644366    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerDied","Data":"998d7939fa237a077fcec64673e75b65d915056e52cd9d42d1c67c5bc05a88aa"}
Oct 30 13:45:04.649423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.649395    4291 generic.go:356] "Generic (PLEG): container finished" podID="b4206617-2601-491e-a274-cb89b62cc103" containerID="60c385250faee45c69234475c2c0b5a7a32cbfa68a12a2b88ecdac60ce0bf287" exitCode=60
Oct 30 13:45:04.653672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.649471    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" event={"ID":"b4206617-2601-491e-a274-cb89b62cc103","Type":"ContainerDied","Data":"60c385250faee45c69234475c2c0b5a7a32cbfa68a12a2b88ecdac60ce0bf287"}
Oct 30 13:45:04.653672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.649764    4291 scope.go:122] "RemoveContainer" containerID="60c385250faee45c69234475c2c0b5a7a32cbfa68a12a2b88ecdac60ce0bf287"
Oct 30 13:45:04.684831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.684798    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" event={"ID":"81d67356-8625-4b93-88fe-b33f21dc9b8e","Type":"ContainerStarted","Data":"2d8bdf1ff9993ab8547d0efd043c268f9e872142e22708984c00b5bce427009a"}
Oct 30 13:45:04.786786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:04.785978    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" podStartSLOduration=4.785958343 podStartE2EDuration="4.785958343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:04.681014572 +0000 UTC m=+2514.941252809" watchObservedRunningTime="2026-10-30 13:45:04.785958343 +0000 UTC m=+2515.046196576"
Oct 30 13:45:05.390466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.360343    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:05.390466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.360398    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:05.715926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.714256    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerStarted","Data":"c02e90e0267c4fb6b93722d0822b4ff512e87bb5e4eee144ee8151066807fecc"}
Oct 30 13:45:05.717702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.717671    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerStarted","Data":"cec501c27d2b46a74e7139498dc5dfc35f36ea388ee388a4581549fe08f8e84f"}
Oct 30 13:45:05.721049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.721023    4291 generic.go:356] "Generic (PLEG): container finished" podID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerID="572cd39f2ba2765522583813824495dbf38742c08ac64459803be71ae4f06038" exitCode=60
Oct 30 13:45:05.721159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.721093    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" event={"ID":"73b26525-0a60-471e-8fe4-2f5a9696b700","Type":"ContainerDied","Data":"572cd39f2ba2765522583813824495dbf38742c08ac64459803be71ae4f06038"}
Oct 30 13:45:05.721159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.721129    4291 scope.go:122] "RemoveContainer" containerID="07db47080aa63de73eca9ce6e701390986b5f5d587cccc39ef952e06c5970128"
Oct 30 13:45:05.721706 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.721681    4291 scope.go:122] "RemoveContainer" containerID="572cd39f2ba2765522583813824495dbf38742c08ac64459803be71ae4f06038"
Oct 30 13:45:05.722084 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.722059    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9_odh-ai-gateway-infra(73b26525-0a60-471e-8fe4-2f5a9696b700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700"
Oct 30 13:45:05.727695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.727671    4291 generic.go:356] "Generic (PLEG): container finished" podID="392207f2-fca4-4e13-80b4-218ff561c46f" containerID="88ad46807e785038c323dc73b7a0857e83f33f1ced3d985eca425f78949b9679" exitCode=60
Oct 30 13:45:05.728280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.728171    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" event={"ID":"392207f2-fca4-4e13-80b4-218ff561c46f","Type":"ContainerDied","Data":"88ad46807e785038c323dc73b7a0857e83f33f1ced3d985eca425f78949b9679"}
Oct 30 13:45:05.728726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.728706    4291 scope.go:122] "RemoveContainer" containerID="88ad46807e785038c323dc73b7a0857e83f33f1ced3d985eca425f78949b9679"
Oct 30 13:45:05.729596 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.729571    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4_odh-ai-gateway-infra(392207f2-fca4-4e13-80b4-218ff561c46f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" podUID="392207f2-fca4-4e13-80b4-218ff561c46f"
Oct 30 13:45:05.730807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.730786    4291 scope.go:122] "RemoveContainer" containerID="ad148db728c74f34c8468c6aeae6384b57ee490c3fbcc3383db74938b18fda11"
Oct 30 13:45:05.731069 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.731047    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz_odh-ai-gateway-infra(28a7ba16-b61c-4139-b1a1-fa2de8b12fa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4"
Oct 30 13:45:05.733394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.733373    4291 generic.go:356] "Generic (PLEG): container finished" podID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerID="2d8bdf1ff9993ab8547d0efd043c268f9e872142e22708984c00b5bce427009a" exitCode=60
Oct 30 13:45:05.733662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.733618    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" event={"ID":"81d67356-8625-4b93-88fe-b33f21dc9b8e","Type":"ContainerDied","Data":"2d8bdf1ff9993ab8547d0efd043c268f9e872142e22708984c00b5bce427009a"}
Oct 30 13:45:05.734065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.734041    4291 scope.go:122] "RemoveContainer" containerID="2d8bdf1ff9993ab8547d0efd043c268f9e872142e22708984c00b5bce427009a"
Oct 30 13:45:05.734381 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.734358    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf_odh-ai-gateway-infra(81d67356-8625-4b93-88fe-b33f21dc9b8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e"
Oct 30 13:45:05.736287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.736262    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerStarted","Data":"a0f5c54a70d91b37e8816cc4ea33520ccca9837ee24a7dd3f94dd87baed24392"}
Oct 30 13:45:05.739707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.739681    4291 generic.go:356] "Generic (PLEG): container finished" podID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerID="43ad622d0989d24612e1d19e4268a260ab5a7d90abd36670f2bd98707b14d74c" exitCode=60
Oct 30 13:45:05.739917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.739896    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" event={"ID":"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b","Type":"ContainerDied","Data":"43ad622d0989d24612e1d19e4268a260ab5a7d90abd36670f2bd98707b14d74c"}
Oct 30 13:45:05.740459 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.740436    4291 scope.go:122] "RemoveContainer" containerID="43ad622d0989d24612e1d19e4268a260ab5a7d90abd36670f2bd98707b14d74c"
Oct 30 13:45:05.740928 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.740906    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx_odh-ai-gateway-infra(bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b"
Oct 30 13:45:05.743397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.743374    4291 generic.go:356] "Generic (PLEG): container finished" podID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerID="7efe4e17226279762c8f656d1768a56e5de80530f11d8e52e75cece08a66a8fb" exitCode=60
Oct 30 13:45:05.744054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.743926    4291 scope.go:122] "RemoveContainer" containerID="7efe4e17226279762c8f656d1768a56e5de80530f11d8e52e75cece08a66a8fb"
Oct 30 13:45:05.744376 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.743945    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" event={"ID":"5714a3ae-1af2-43fd-824c-e6536556a50f","Type":"ContainerDied","Data":"7efe4e17226279762c8f656d1768a56e5de80530f11d8e52e75cece08a66a8fb"}
Oct 30 13:45:05.749969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.748506    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd06531d-fb03-48a0-864d-b3e33da93784" containerID="a4975d30345c5db66c695eeacd7eee32e99d2cfd090cc152c1e2a1b37b4ae8a0" exitCode=60
Oct 30 13:45:05.749969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.748630    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" event={"ID":"fd06531d-fb03-48a0-864d-b3e33da93784","Type":"ContainerDied","Data":"a4975d30345c5db66c695eeacd7eee32e99d2cfd090cc152c1e2a1b37b4ae8a0"}
Oct 30 13:45:05.749969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.749292    4291 scope.go:122] "RemoveContainer" containerID="a4975d30345c5db66c695eeacd7eee32e99d2cfd090cc152c1e2a1b37b4ae8a0"
Oct 30 13:45:05.750379 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.750305    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-mt5wq_odh-ai-gateway-infra(fd06531d-fb03-48a0-864d-b3e33da93784)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" podUID="fd06531d-fb03-48a0-864d-b3e33da93784"
Oct 30 13:45:05.758118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.758079    4291 generic.go:356] "Generic (PLEG): container finished" podID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerID="97e26caa2149a2bd9195d3bbd1f8ef97b9d813bee70414b1b5c561e9006d2261" exitCode=60
Oct 30 13:45:05.758444 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.758415    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" event={"ID":"696f6bbf-ec44-4546-8d71-29bd05cf026e","Type":"ContainerDied","Data":"97e26caa2149a2bd9195d3bbd1f8ef97b9d813bee70414b1b5c561e9006d2261"}
Oct 30 13:45:05.759176 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.759154    4291 scope.go:122] "RemoveContainer" containerID="97e26caa2149a2bd9195d3bbd1f8ef97b9d813bee70414b1b5c561e9006d2261"
Oct 30 13:45:05.759859 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.759830    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz_odh-ai-gateway-infra(696f6bbf-ec44-4546-8d71-29bd05cf026e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e"
Oct 30 13:45:05.762960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.762933    4291 generic.go:356] "Generic (PLEG): container finished" podID="80678d19-5845-487b-9329-577e2cf9738e" containerID="81353d5f80abbbaf30cb3594e72d600e81db86790df954bb4a91044a36becc58" exitCode=60
Oct 30 13:45:05.763266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.763244    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerDied","Data":"81353d5f80abbbaf30cb3594e72d600e81db86790df954bb4a91044a36becc58"}
Oct 30 13:45:05.763878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.763857    4291 scope.go:122] "RemoveContainer" containerID="81353d5f80abbbaf30cb3594e72d600e81db86790df954bb4a91044a36becc58"
Oct 30 13:45:05.816256 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.816204    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw_odh-ai-gateway-infra(80678d19-5845-487b-9329-577e2cf9738e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" podUID="80678d19-5845-487b-9329-577e2cf9738e"
Oct 30 13:45:05.830034 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.797806    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d_odh-ai-gateway-infra(5714a3ae-1af2-43fd-824c-e6536556a50f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f"
Oct 30 13:45:05.830697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.819032    4291 scope.go:122] "RemoveContainer" containerID="262f0ca81d29d52c307b4d527b5e8688f27adab2e505300b65a22b9972ebcfee"
Oct 30 13:45:05.830946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:05.830930    4291 scope.go:122] "RemoveContainer" containerID="c3baf89fc92dcd79252a477d3210cf31539b3b49da39375dbc330c1ea1100170"
Oct 30 13:45:05.863501 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:05.863445    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz_odh-ai-gateway-infra(5ec89a7d-eb27-48de-a793-86a7b0d5389b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b"
Oct 30 13:45:06.049943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.031022    4291 scope.go:122] "RemoveContainer" containerID="015ed6f4c4b91fb7d5050ed189edfadb317b298493107a0fa270a225fa84b826"
Oct 30 13:45:06.280867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.280834    4291 scope.go:122] "RemoveContainer" containerID="77d5c2f037d2951eb1c50f9b2334629ced06dbfdd4f519c92f90d4b61baf226b"
Oct 30 13:45:06.369293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.369254    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:06.369530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.369503    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:06.559985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.547942    4291 scope.go:122] "RemoveContainer" containerID="eed00361a53aa37fb919b67919bf4e7c2d691009044264978f14f1b932c8985d"
Oct 30 13:45:06.687329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.686843    4291 scope.go:122] "RemoveContainer" containerID="c97361b0ffde98748dcb8624dce4605da0b97da2c7fd71917b08256cf33c986f"
Oct 30 13:45:06.729027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.728784    4291 scope.go:122] "RemoveContainer" containerID="c4179efa965cf9062a4a2fec0c57dd6c8d7ebc8981c464738d4a06cf905690be"
Oct 30 13:45:06.743671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.743629    4291 scope.go:122] "RemoveContainer" containerID="e109526449e6586066a75e3364bdc1e10c7ae46edb2a38591386cb4fbba9d2d5"
Oct 30 13:45:06.838515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.838422    4291 generic.go:356] "Generic (PLEG): container finished" podID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerID="c02e90e0267c4fb6b93722d0822b4ff512e87bb5e4eee144ee8151066807fecc" exitCode=60
Oct 30 13:45:06.838515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.838498    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerDied","Data":"c02e90e0267c4fb6b93722d0822b4ff512e87bb5e4eee144ee8151066807fecc"}
Oct 30 13:45:06.838952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.838932    4291 scope.go:122] "RemoveContainer" containerID="c02e90e0267c4fb6b93722d0822b4ff512e87bb5e4eee144ee8151066807fecc"
Oct 30 13:45:06.839283 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.839263    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr_odh-ai-gateway-infra(75451d0a-8ddb-411c-98c7-b4fa2101d048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048"
Oct 30 13:45:06.842316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.842291    4291 generic.go:356] "Generic (PLEG): container finished" podID="7253bff1-99f9-4619-9239-3a76834a05dc" containerID="cec501c27d2b46a74e7139498dc5dfc35f36ea388ee388a4581549fe08f8e84f" exitCode=60
Oct 30 13:45:06.842460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.842430    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerDied","Data":"cec501c27d2b46a74e7139498dc5dfc35f36ea388ee388a4581549fe08f8e84f"}
Oct 30 13:45:06.843299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.843276    4291 scope.go:122] "RemoveContainer" containerID="cec501c27d2b46a74e7139498dc5dfc35f36ea388ee388a4581549fe08f8e84f"
Oct 30 13:45:06.843727 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.843698    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc_odh-ai-gateway-infra(7253bff1-99f9-4619-9239-3a76834a05dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" podUID="7253bff1-99f9-4619-9239-3a76834a05dc"
Oct 30 13:45:06.845019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.844999    4291 scope.go:122] "RemoveContainer" containerID="572cd39f2ba2765522583813824495dbf38742c08ac64459803be71ae4f06038"
Oct 30 13:45:06.845352 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.845325    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9_odh-ai-gateway-infra(73b26525-0a60-471e-8fe4-2f5a9696b700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700"
Oct 30 13:45:06.846769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.846749    4291 scope.go:122] "RemoveContainer" containerID="88ad46807e785038c323dc73b7a0857e83f33f1ced3d985eca425f78949b9679"
Oct 30 13:45:06.847065 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.847038    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4_odh-ai-gateway-infra(392207f2-fca4-4e13-80b4-218ff561c46f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" podUID="392207f2-fca4-4e13-80b4-218ff561c46f"
Oct 30 13:45:06.848475 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.848449    4291 generic.go:356] "Generic (PLEG): container finished" podID="32860ef1-4fcb-45f2-8297-f3330e738442" containerID="5c337e9ffacdbceadec1c617680a9351ae08d78c8e67a0b2638d755c6b6ee3f1" exitCode=60
Oct 30 13:45:06.848565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.848506    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" event={"ID":"32860ef1-4fcb-45f2-8297-f3330e738442","Type":"ContainerDied","Data":"5c337e9ffacdbceadec1c617680a9351ae08d78c8e67a0b2638d755c6b6ee3f1"}
Oct 30 13:45:06.848771 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.848749    4291 scope.go:122] "RemoveContainer" containerID="5c337e9ffacdbceadec1c617680a9351ae08d78c8e67a0b2638d755c6b6ee3f1"
Oct 30 13:45:06.848996 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.848977    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq_odh-ai-gateway-infra(32860ef1-4fcb-45f2-8297-f3330e738442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" podUID="32860ef1-4fcb-45f2-8297-f3330e738442"
Oct 30 13:45:06.876658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.875401    4291 generic.go:356] "Generic (PLEG): container finished" podID="46831069-b252-4c2d-87d4-834789ce4d6e" containerID="f6b3d54c540d40dcc27d6feda01470da8899ed6b94867b46758d29cd762d9648" exitCode=60
Oct 30 13:45:06.899661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.881175    4291 scope.go:122] "RemoveContainer" containerID="f6b3d54c540d40dcc27d6feda01470da8899ed6b94867b46758d29cd762d9648"
Oct 30 13:45:06.899661 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.881602    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499_odh-ai-gateway-infra(46831069-b252-4c2d-87d4-834789ce4d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" podUID="46831069-b252-4c2d-87d4-834789ce4d6e"
Oct 30 13:45:06.899661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.875507    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" event={"ID":"46831069-b252-4c2d-87d4-834789ce4d6e","Type":"ContainerDied","Data":"f6b3d54c540d40dcc27d6feda01470da8899ed6b94867b46758d29cd762d9648"}
Oct 30 13:45:06.909213 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.909182    4291 scope.go:122] "RemoveContainer" containerID="8f48a7494083c80eed6d4dd58344d3f6d0fb84407a08eaefdab8676aac029273"
Oct 30 13:45:06.913146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.913115    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerStarted","Data":"e2105aa159c9dcfc6dbc92aa73422e9e61f37f07a853d1e6bfc3f3f4692365ff"}
Oct 30 13:45:06.920956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.920935    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" event={"ID":"b4206617-2601-491e-a274-cb89b62cc103","Type":"ContainerStarted","Data":"5a006298f66fbbb47fb03d26a3a5e821f1243fe998d595d127fa72f0bd6516fc"}
Oct 30 13:45:06.921146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.921117    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerDied","Data":"a0f5c54a70d91b37e8816cc4ea33520ccca9837ee24a7dd3f94dd87baed24392"}
Oct 30 13:45:06.921270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.918436    4291 scope.go:122] "RemoveContainer" containerID="2d8bdf1ff9993ab8547d0efd043c268f9e872142e22708984c00b5bce427009a"
Oct 30 13:45:06.921370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.920451    4291 generic.go:356] "Generic (PLEG): container finished" podID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerID="a0f5c54a70d91b37e8816cc4ea33520ccca9837ee24a7dd3f94dd87baed24392" exitCode=60
Oct 30 13:45:06.921454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.921442    4291 scope.go:122] "RemoveContainer" containerID="a0f5c54a70d91b37e8816cc4ea33520ccca9837ee24a7dd3f94dd87baed24392"
Oct 30 13:45:06.921893 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.921871    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29_odh-ai-gateway-infra(ba738823-8bdf-4597-8eed-7e3a3fb027f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2"
Oct 30 13:45:06.923681 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.923659    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf_odh-ai-gateway-infra(81d67356-8625-4b93-88fe-b33f21dc9b8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e"
Oct 30 13:45:06.924344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.924261    4291 scope.go:122] "RemoveContainer" containerID="43ad622d0989d24612e1d19e4268a260ab5a7d90abd36670f2bd98707b14d74c"
Oct 30 13:45:06.925059 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.925033    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx_odh-ai-gateway-infra(bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b"
Oct 30 13:45:06.927447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.927428    4291 scope.go:122] "RemoveContainer" containerID="7efe4e17226279762c8f656d1768a56e5de80530f11d8e52e75cece08a66a8fb"
Oct 30 13:45:06.927842 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.927820    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d_odh-ai-gateway-infra(5714a3ae-1af2-43fd-824c-e6536556a50f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f"
Oct 30 13:45:06.929464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.929444    4291 scope.go:122] "RemoveContainer" containerID="a4975d30345c5db66c695eeacd7eee32e99d2cfd090cc152c1e2a1b37b4ae8a0"
Oct 30 13:45:06.929848 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.929826    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-mt5wq_odh-ai-gateway-infra(fd06531d-fb03-48a0-864d-b3e33da93784)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" podUID="fd06531d-fb03-48a0-864d-b3e33da93784"
Oct 30 13:45:06.932299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.932282    4291 scope.go:122] "RemoveContainer" containerID="81353d5f80abbbaf30cb3594e72d600e81db86790df954bb4a91044a36becc58"
Oct 30 13:45:06.933068 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.933043    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw_odh-ai-gateway-infra(80678d19-5845-487b-9329-577e2cf9738e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" podUID="80678d19-5845-487b-9329-577e2cf9738e"
Oct 30 13:45:06.933196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.933182    4291 scope.go:122] "RemoveContainer" containerID="97e26caa2149a2bd9195d3bbd1f8ef97b9d813bee70414b1b5c561e9006d2261"
Oct 30 13:45:06.933695 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:06.933671    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz_odh-ai-gateway-infra(696f6bbf-ec44-4546-8d71-29bd05cf026e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e"
Oct 30 13:45:06.944071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:06.944032    4291 scope.go:122] "RemoveContainer" containerID="055cc1b3e2ba9d8010ec736fd8b0c8cf6573ca3dcf49bf44002274f26311a3ad"
Oct 30 13:45:07.130728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.128098    4291 scope.go:122] "RemoveContainer" containerID="7b501803741619e343054c7f8b7435b4b0ba3feb7eb8e6a5496f65bc56d11626"
Oct 30 13:45:07.147518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.147485    4291 scope.go:122] "RemoveContainer" containerID="e21fc35d31ce2725ae24b64fff7e5bf0d4a484a2f5b6d0d0bfcfeb52c3e5e681"
Oct 30 13:45:07.237135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.237107    4291 scope.go:122] "RemoveContainer" containerID="380f3ce46f1c1b30bf34d272bed9db6894144d217a26892fe8fd1006cf31db5e"
Oct 30 13:45:07.355025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.354930    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:07.355278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.355250    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:07.941798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.941769    4291 scope.go:122] "RemoveContainer" containerID="cec501c27d2b46a74e7139498dc5dfc35f36ea388ee388a4581549fe08f8e84f"
Oct 30 13:45:07.942241 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:07.942172    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc_odh-ai-gateway-infra(7253bff1-99f9-4619-9239-3a76834a05dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" podUID="7253bff1-99f9-4619-9239-3a76834a05dc"
Oct 30 13:45:07.943619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.943597    4291 scope.go:122] "RemoveContainer" containerID="5c337e9ffacdbceadec1c617680a9351ae08d78c8e67a0b2638d755c6b6ee3f1"
Oct 30 13:45:07.944104 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:07.944065    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq_odh-ai-gateway-infra(32860ef1-4fcb-45f2-8297-f3330e738442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" podUID="32860ef1-4fcb-45f2-8297-f3330e738442"
Oct 30 13:45:07.945455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.945430    4291 scope.go:122] "RemoveContainer" containerID="f6b3d54c540d40dcc27d6feda01470da8899ed6b94867b46758d29cd762d9648"
Oct 30 13:45:07.945812 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:07.945786    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499_odh-ai-gateway-infra(46831069-b252-4c2d-87d4-834789ce4d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" podUID="46831069-b252-4c2d-87d4-834789ce4d6e"
Oct 30 13:45:07.947018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.946995    4291 generic.go:356] "Generic (PLEG): container finished" podID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerID="e2105aa159c9dcfc6dbc92aa73422e9e61f37f07a853d1e6bfc3f3f4692365ff" exitCode=60
Oct 30 13:45:07.947108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.947064    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerDied","Data":"e2105aa159c9dcfc6dbc92aa73422e9e61f37f07a853d1e6bfc3f3f4692365ff"}
Oct 30 13:45:07.947168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.947122    4291 scope.go:122] "RemoveContainer" containerID="998d7939fa237a077fcec64673e75b65d915056e52cd9d42d1c67c5bc05a88aa"
Oct 30 13:45:07.947323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.947306    4291 scope.go:122] "RemoveContainer" containerID="e2105aa159c9dcfc6dbc92aa73422e9e61f37f07a853d1e6bfc3f3f4692365ff"
Oct 30 13:45:07.947673 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:07.947650    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848_odh-ai-gateway-infra(5aaca5e6-383c-46b3-84b5-b5aa882e37c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0"
Oct 30 13:45:07.949293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.949271    4291 generic.go:356] "Generic (PLEG): container finished" podID="b4206617-2601-491e-a274-cb89b62cc103" containerID="5a006298f66fbbb47fb03d26a3a5e821f1243fe998d595d127fa72f0bd6516fc" exitCode=60
Oct 30 13:45:07.949372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.949347    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" event={"ID":"b4206617-2601-491e-a274-cb89b62cc103","Type":"ContainerDied","Data":"5a006298f66fbbb47fb03d26a3a5e821f1243fe998d595d127fa72f0bd6516fc"}
Oct 30 13:45:07.949600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:07.949586    4291 scope.go:122] "RemoveContainer" containerID="5a006298f66fbbb47fb03d26a3a5e821f1243fe998d595d127fa72f0bd6516fc"
Oct 30 13:45:07.971663 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:07.959709    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn_odh-ai-gateway-infra(b4206617-2601-491e-a274-cb89b62cc103)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" podUID="b4206617-2601-491e-a274-cb89b62cc103"
Oct 30 13:45:08.002818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:08.002782    4291 scope.go:122] "RemoveContainer" containerID="a0f5c54a70d91b37e8816cc4ea33520ccca9837ee24a7dd3f94dd87baed24392"
Oct 30 13:45:08.013496 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:08.013420    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29_odh-ai-gateway-infra(ba738823-8bdf-4597-8eed-7e3a3fb027f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2"
Oct 30 13:45:08.016975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:08.016948    4291 scope.go:122] "RemoveContainer" containerID="60c385250faee45c69234475c2c0b5a7a32cbfa68a12a2b88ecdac60ce0bf287"
Oct 30 13:45:08.020884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:08.020862    4291 scope.go:122] "RemoveContainer" containerID="c02e90e0267c4fb6b93722d0822b4ff512e87bb5e4eee144ee8151066807fecc"
Oct 30 13:45:08.021130 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:08.021108    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr_odh-ai-gateway-infra(75451d0a-8ddb-411c-98c7-b4fa2101d048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048"
Oct 30 13:45:08.250062 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:08.246429    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:45:08.384662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:08.374105    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:08.384662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:08.374169    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:09.035026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:09.034993    4291 scope.go:122] "RemoveContainer" containerID="e2105aa159c9dcfc6dbc92aa73422e9e61f37f07a853d1e6bfc3f3f4692365ff"
Oct 30 13:45:09.035436 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:09.035395    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848_odh-ai-gateway-infra(5aaca5e6-383c-46b3-84b5-b5aa882e37c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0"
Oct 30 13:45:09.036738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:09.036716    4291 scope.go:122] "RemoveContainer" containerID="5a006298f66fbbb47fb03d26a3a5e821f1243fe998d595d127fa72f0bd6516fc"
Oct 30 13:45:09.037013 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:09.036989    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn_odh-ai-gateway-infra(b4206617-2601-491e-a274-cb89b62cc103)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" podUID="b4206617-2601-491e-a274-cb89b62cc103"
Oct 30 13:45:09.362093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:09.362050    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:09.362420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:09.362384    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:10.351896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:10.351856    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:10.352556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:10.351918    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:11.351385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:11.351342    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused" start-of-body=
Oct 30 13:45:11.351584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:11.351426    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": dial tcp 10.133.1.0:15021: connect: connection refused"
Oct 30 13:45:13.392884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:13.392833    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:45:13.393351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:13.392928    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:14.394275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:14.394235    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-pbpdb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:45:14.394754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:14.394309    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.0:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:45:14.462069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:14.462031    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:14.462751 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:14.462693    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:14.463693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:14.463663    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:45:15.013115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:15.013076    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"]
Oct 30 13:45:15.883083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:15.881031    4291 scope.go:122] "RemoveContainer" containerID="ad148db728c74f34c8468c6aeae6384b57ee490c3fbcc3383db74938b18fda11"
Oct 30 13:45:16.871393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:16.871358    4291 scope.go:122] "RemoveContainer" containerID="c3baf89fc92dcd79252a477d3210cf31539b3b49da39375dbc330c1ea1100170"
Oct 30 13:45:16.871616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:16.871530    4291 scope.go:122] "RemoveContainer" containerID="88ad46807e785038c323dc73b7a0857e83f33f1ced3d985eca425f78949b9679"
Oct 30 13:45:17.141253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.141103    4291 generic.go:356] "Generic (PLEG): container finished" podID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerID="3be8f7e89b7167817bbcebe3b4762094e132b1ca6600300571a6b849efaa9d8b" exitCode=60
Oct 30 13:45:17.141702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.141277    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" event={"ID":"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4","Type":"ContainerDied","Data":"3be8f7e89b7167817bbcebe3b4762094e132b1ca6600300571a6b849efaa9d8b"}
Oct 30 13:45:17.141702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.141343    4291 scope.go:122] "RemoveContainer" containerID="ad148db728c74f34c8468c6aeae6384b57ee490c3fbcc3383db74938b18fda11"
Oct 30 13:45:17.141988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.141968    4291 scope.go:122] "RemoveContainer" containerID="3be8f7e89b7167817bbcebe3b4762094e132b1ca6600300571a6b849efaa9d8b"
Oct 30 13:45:17.142425 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:17.142395    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz_odh-ai-gateway-infra(28a7ba16-b61c-4139-b1a1-fa2de8b12fa4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4"
Oct 30 13:45:17.871833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.871204    4291 scope.go:122] "RemoveContainer" containerID="97e26caa2149a2bd9195d3bbd1f8ef97b9d813bee70414b1b5c561e9006d2261"
Oct 30 13:45:17.871833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.871358    4291 scope.go:122] "RemoveContainer" containerID="a4975d30345c5db66c695eeacd7eee32e99d2cfd090cc152c1e2a1b37b4ae8a0"
Oct 30 13:45:17.871833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.871459    4291 scope.go:122] "RemoveContainer" containerID="43ad622d0989d24612e1d19e4268a260ab5a7d90abd36670f2bd98707b14d74c"
Oct 30 13:45:17.871833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.871526    4291 scope.go:122] "RemoveContainer" containerID="c02e90e0267c4fb6b93722d0822b4ff512e87bb5e4eee144ee8151066807fecc"
Oct 30 13:45:17.872215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:17.872168    4291 scope.go:122] "RemoveContainer" containerID="81353d5f80abbbaf30cb3594e72d600e81db86790df954bb4a91044a36becc58"
Oct 30 13:45:18.153625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.153534    4291 generic.go:356] "Generic (PLEG): container finished" podID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerID="aa95a27b32f84e2cb1cf6a894842da7081e4a650e125b43fb351d3ed0326b857" exitCode=60
Oct 30 13:45:18.153625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.153602    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" event={"ID":"5ec89a7d-eb27-48de-a793-86a7b0d5389b","Type":"ContainerDied","Data":"aa95a27b32f84e2cb1cf6a894842da7081e4a650e125b43fb351d3ed0326b857"}
Oct 30 13:45:18.154199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.153678    4291 scope.go:122] "RemoveContainer" containerID="c3baf89fc92dcd79252a477d3210cf31539b3b49da39375dbc330c1ea1100170"
Oct 30 13:45:18.154199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.154064    4291 scope.go:122] "RemoveContainer" containerID="aa95a27b32f84e2cb1cf6a894842da7081e4a650e125b43fb351d3ed0326b857"
Oct 30 13:45:18.154373 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:18.154347    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz_odh-ai-gateway-infra(5ec89a7d-eb27-48de-a793-86a7b0d5389b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b"
Oct 30 13:45:18.156012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.155991    4291 generic.go:356] "Generic (PLEG): container finished" podID="392207f2-fca4-4e13-80b4-218ff561c46f" containerID="38fc631231ebf3d4200e379029a9ec0d0d30145563c137c623d9176394a816a0" exitCode=60
Oct 30 13:45:18.156141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.156036    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" event={"ID":"392207f2-fca4-4e13-80b4-218ff561c46f","Type":"ContainerDied","Data":"38fc631231ebf3d4200e379029a9ec0d0d30145563c137c623d9176394a816a0"}
Oct 30 13:45:18.156434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.156411    4291 scope.go:122] "RemoveContainer" containerID="38fc631231ebf3d4200e379029a9ec0d0d30145563c137c623d9176394a816a0"
Oct 30 13:45:18.156717 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:18.156697    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4_odh-ai-gateway-infra(392207f2-fca4-4e13-80b4-218ff561c46f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" podUID="392207f2-fca4-4e13-80b4-218ff561c46f"
Oct 30 13:45:18.215741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.215708    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"]
Oct 30 13:45:18.278727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.274802    4291 scope.go:122] "RemoveContainer" containerID="88ad46807e785038c323dc73b7a0857e83f33f1ced3d985eca425f78949b9679"
Oct 30 13:45:18.491754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.491714    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:18.537674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.533716    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-maas-api-service-ca\") pod \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") "
Oct 30 13:45:18.537674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.533812    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-kube-api-access-lrn6d\" (UniqueName: \"kubernetes.io/projected/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-kube-api-access-lrn6d\") pod \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\" (UID: \"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4\") "
Oct 30 13:45:18.550669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.550092    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-maas-api-service-ca" pod "28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" (UID: "28a7ba16-b61c-4139-b1a1-fa2de8b12fa4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:18.553668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.551836    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-kube-api-access-lrn6d" pod "28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" (UID: "28a7ba16-b61c-4139-b1a1-fa2de8b12fa4"). InnerVolumeSpecName "kube-api-access-lrn6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:18.635696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.635646    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:18.635696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.635681    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrn6d\" (UniqueName: \"kubernetes.io/projected/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4-kube-api-access-lrn6d\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:18.872032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.871978    4291 scope.go:122] "RemoveContainer" containerID="a0f5c54a70d91b37e8816cc4ea33520ccca9837ee24a7dd3f94dd87baed24392"
Oct 30 13:45:18.872255 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.872118    4291 scope.go:122] "RemoveContainer" containerID="572cd39f2ba2765522583813824495dbf38742c08ac64459803be71ae4f06038"
Oct 30 13:45:18.873100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.872701    4291 scope.go:122] "RemoveContainer" containerID="e2105aa159c9dcfc6dbc92aa73422e9e61f37f07a853d1e6bfc3f3f4692365ff"
Oct 30 13:45:18.873100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.872803    4291 scope.go:122] "RemoveContainer" containerID="5a006298f66fbbb47fb03d26a3a5e821f1243fe998d595d127fa72f0bd6516fc"
Oct 30 13:45:18.873100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.872916    4291 scope.go:122] "RemoveContainer" containerID="7efe4e17226279762c8f656d1768a56e5de80530f11d8e52e75cece08a66a8fb"
Oct 30 13:45:18.873100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.872973    4291 scope.go:122] "RemoveContainer" containerID="f6b3d54c540d40dcc27d6feda01470da8899ed6b94867b46758d29cd762d9648"
Oct 30 13:45:18.873328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:18.873205    4291 scope.go:122] "RemoveContainer" containerID="2d8bdf1ff9993ab8547d0efd043c268f9e872142e22708984c00b5bce427009a"
Oct 30 13:45:19.171856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.171824    4291 generic.go:356] "Generic (PLEG): container finished" podID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerID="c0089aca5ad5bf7c1d40171def1a6a4fbe60a8c62c22b2d0cd5338dea57b2691" exitCode=60
Oct 30 13:45:19.172390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.171892    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" event={"ID":"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b","Type":"ContainerDied","Data":"c0089aca5ad5bf7c1d40171def1a6a4fbe60a8c62c22b2d0cd5338dea57b2691"}
Oct 30 13:45:19.172390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.171951    4291 scope.go:122] "RemoveContainer" containerID="43ad622d0989d24612e1d19e4268a260ab5a7d90abd36670f2bd98707b14d74c"
Oct 30 13:45:19.172491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.172409    4291 scope.go:122] "RemoveContainer" containerID="c0089aca5ad5bf7c1d40171def1a6a4fbe60a8c62c22b2d0cd5338dea57b2691"
Oct 30 13:45:19.172899 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:19.172872    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx_odh-ai-gateway-infra(bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b"
Oct 30 13:45:19.174868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.174829    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" event={"ID":"fd06531d-fb03-48a0-864d-b3e33da93784","Type":"ContainerStarted","Data":"07605e29ca04db74af2396bfe0b1e1b01d7040f3825b98698e13d70b8dd9dc78"}
Oct 30 13:45:19.177432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.177389    4291 generic.go:356] "Generic (PLEG): container finished" podID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerID="cb837c67c229077fa5e60d2e93f511d4d55aab0141ad0c1d8c10f572e3982b21" exitCode=60
Oct 30 13:45:19.177713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.177464    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" event={"ID":"696f6bbf-ec44-4546-8d71-29bd05cf026e","Type":"ContainerDied","Data":"cb837c67c229077fa5e60d2e93f511d4d55aab0141ad0c1d8c10f572e3982b21"}
Oct 30 13:45:19.178071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.178050    4291 scope.go:122] "RemoveContainer" containerID="cb837c67c229077fa5e60d2e93f511d4d55aab0141ad0c1d8c10f572e3982b21"
Oct 30 13:45:19.178517 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:19.178491    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz_odh-ai-gateway-infra(696f6bbf-ec44-4546-8d71-29bd05cf026e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e"
Oct 30 13:45:19.180329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.180294    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerStarted","Data":"63a845b891e13d2138ff980a633b2a496992beecfa55e7d8f84ac571a259282e"}
Oct 30 13:45:19.180883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.180858    4291 scope.go:122] "RemoveContainer" containerID="63a845b891e13d2138ff980a633b2a496992beecfa55e7d8f84ac571a259282e"
Oct 30 13:45:19.181346 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:19.181310    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw_odh-ai-gateway-infra(80678d19-5845-487b-9329-577e2cf9738e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" podUID="80678d19-5845-487b-9329-577e2cf9738e"
Oct 30 13:45:19.185079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.185029    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerStarted","Data":"fea266ac2b58a80c350da01e32d48d1c96990490597ca6d882b3bb2f236dde25"}
Oct 30 13:45:19.188774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.188734    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"
Oct 30 13:45:19.188774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.188761    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz" event={"ID":"28a7ba16-b61c-4139-b1a1-fa2de8b12fa4","Type":"ContainerDied","Data":"c868f9a5b12bded6ccdbb3a5fed3381d553c359e30ed3c72c8547cf993d4062b"}
Oct 30 13:45:19.207952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.207919    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"]
Oct 30 13:45:19.261907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.261336    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"]
Oct 30 13:45:19.318251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.318182    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" podStartSLOduration=19.31816345 podStartE2EDuration="19.31816345s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:45:19.311251449 +0000 UTC m=+2529.571489684" watchObservedRunningTime="2026-10-30 13:45:19.31816345 +0000 UTC m=+2529.578401685"
Oct 30 13:45:19.417407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.417331    4291 scope.go:122] "RemoveContainer" containerID="97e26caa2149a2bd9195d3bbd1f8ef97b9d813bee70414b1b5c561e9006d2261"
Oct 30 13:45:19.694470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.694389    4291 scope.go:122] "RemoveContainer" containerID="3be8f7e89b7167817bbcebe3b4762094e132b1ca6600300571a6b849efaa9d8b"
Oct 30 13:45:19.696961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.696938    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:19.700310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.700271    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:19.726503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.726470    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"]
Oct 30 13:45:19.731095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.731068    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889465-bh7sz"]
Oct 30 13:45:19.760528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.760492    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ec89a7d-eb27-48de-a793-86a7b0d5389b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89a7d-eb27-48de-a793-86a7b0d5389b-maas-api-service-ca\") pod \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") "
Oct 30 13:45:19.761235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.760820    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/392207f2-fca4-4e13-80b4-218ff561c46f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/392207f2-fca4-4e13-80b4-218ff561c46f-maas-api-service-ca\") pod \"392207f2-fca4-4e13-80b4-218ff561c46f\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") "
Oct 30 13:45:19.761235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.760862    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ec89a7d-eb27-48de-a793-86a7b0d5389b-maas-api-service-ca" pod "5ec89a7d-eb27-48de-a793-86a7b0d5389b" (UID: "5ec89a7d-eb27-48de-a793-86a7b0d5389b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:19.761235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.760879    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ec89a7d-eb27-48de-a793-86a7b0d5389b-kube-api-access-wvfck\" (UniqueName: \"kubernetes.io/projected/5ec89a7d-eb27-48de-a793-86a7b0d5389b-kube-api-access-wvfck\") pod \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\" (UID: \"5ec89a7d-eb27-48de-a793-86a7b0d5389b\") "
Oct 30 13:45:19.761235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.761128    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/392207f2-fca4-4e13-80b4-218ff561c46f-kube-api-access-ld9wc\" (UniqueName: \"kubernetes.io/projected/392207f2-fca4-4e13-80b4-218ff561c46f-kube-api-access-ld9wc\") pod \"392207f2-fca4-4e13-80b4-218ff561c46f\" (UID: \"392207f2-fca4-4e13-80b4-218ff561c46f\") "
Oct 30 13:45:19.761235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.761158    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/392207f2-fca4-4e13-80b4-218ff561c46f-maas-api-service-ca" pod "392207f2-fca4-4e13-80b4-218ff561c46f" (UID: "392207f2-fca4-4e13-80b4-218ff561c46f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:19.761758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.761736    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ec89a7d-eb27-48de-a793-86a7b0d5389b-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.761845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.761761    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/392207f2-fca4-4e13-80b4-218ff561c46f-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.763157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.763130    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ec89a7d-eb27-48de-a793-86a7b0d5389b-kube-api-access-wvfck" pod "5ec89a7d-eb27-48de-a793-86a7b0d5389b" (UID: "5ec89a7d-eb27-48de-a793-86a7b0d5389b"). InnerVolumeSpecName "kube-api-access-wvfck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:19.763402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.763384    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/392207f2-fca4-4e13-80b4-218ff561c46f-kube-api-access-ld9wc" pod "392207f2-fca4-4e13-80b4-218ff561c46f" (UID: "392207f2-fca4-4e13-80b4-218ff561c46f"). InnerVolumeSpecName "kube-api-access-ld9wc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:19.863496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.863454    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld9wc\" (UniqueName: \"kubernetes.io/projected/392207f2-fca4-4e13-80b4-218ff561c46f-kube-api-access-ld9wc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.863496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.863486    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvfck\" (UniqueName: \"kubernetes.io/projected/5ec89a7d-eb27-48de-a793-86a7b0d5389b-kube-api-access-wvfck\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:19.883046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:19.883011    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" path="/var/lib/kubelet/pods/28a7ba16-b61c-4139-b1a1-fa2de8b12fa4/volumes"
Oct 30 13:45:20.013657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.013550    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4" podUID="998880a0-7ee4-452c-89ca-d2cc33a61c7a" containerName="payload-processing" containerID="cri-o://c2d9f0634981e4d6f9221f4822a62628aabfea059c212d4e5d33090ab73c05a8" gracePeriod=25
Oct 30 13:45:20.212476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.211227    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"
Oct 30 13:45:20.217216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.215177    4291 generic.go:356] "Generic (PLEG): container finished" podID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerID="fea266ac2b58a80c350da01e32d48d1c96990490597ca6d882b3bb2f236dde25" exitCode=60
Oct 30 13:45:20.221069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.220388    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"
Oct 30 13:45:20.227087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.211061    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz" event={"ID":"5ec89a7d-eb27-48de-a793-86a7b0d5389b","Type":"ContainerDied","Data":"77f163fe2a669deeba08275a71498fbb7566fc475a1b653161b431299533b4a7"}
Oct 30 13:45:20.227087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.225743    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerDied","Data":"fea266ac2b58a80c350da01e32d48d1c96990490597ca6d882b3bb2f236dde25"}
Oct 30 13:45:20.227087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.225851    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" event={"ID":"73b26525-0a60-471e-8fe4-2f5a9696b700","Type":"ContainerStarted","Data":"2f791c0370295c01934b102104f5f3256df36e94d7f1c140f0bc9aaa95f3346f"}
Oct 30 13:45:20.227087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.225892    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4" event={"ID":"392207f2-fca4-4e13-80b4-218ff561c46f","Type":"ContainerDied","Data":"b27e72d6e2fed62a2a3d78718227bf113afbfa29eedf6a43874c1d8da0a2fd3b"}
Oct 30 13:45:20.227087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.226889    4291 scope.go:122] "RemoveContainer" containerID="aa95a27b32f84e2cb1cf6a894842da7081e4a650e125b43fb351d3ed0326b857"
Oct 30 13:45:20.227469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.227406    4291 scope.go:122] "RemoveContainer" containerID="fea266ac2b58a80c350da01e32d48d1c96990490597ca6d882b3bb2f236dde25"
Oct 30 13:45:20.228711 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:20.227877    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr_odh-ai-gateway-infra(75451d0a-8ddb-411c-98c7-b4fa2101d048)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048"
Oct 30 13:45:20.231471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.230149    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" event={"ID":"46831069-b252-4c2d-87d4-834789ce4d6e","Type":"ContainerStarted","Data":"655a7c61e5a0908d06e67a5acfc6a1f02bd0e7cd73734f1904e170e8f2ab67e8"}
Oct 30 13:45:20.233451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.233424    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerStarted","Data":"4f4b283a89b19149e770f4508d08f7bcbaaf87207b1153bce369e2bfaba24e67"}
Oct 30 13:45:20.236175 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.236143    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"]
Oct 30 13:45:20.237962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.237934    4291 generic.go:356] "Generic (PLEG): container finished" podID="b4206617-2601-491e-a274-cb89b62cc103" containerID="1da92cc55d3edb93f9413c4ba2671208e15dac862b42f3d8a7b4bfd74d27eb3f" exitCode=60
Oct 30 13:45:20.238292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.238270    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" event={"ID":"b4206617-2601-491e-a274-cb89b62cc103","Type":"ContainerDied","Data":"1da92cc55d3edb93f9413c4ba2671208e15dac862b42f3d8a7b4bfd74d27eb3f"}
Oct 30 13:45:20.239370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.239328    4291 scope.go:122] "RemoveContainer" containerID="1da92cc55d3edb93f9413c4ba2671208e15dac862b42f3d8a7b4bfd74d27eb3f"
Oct 30 13:45:20.240081 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:20.240056    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn_odh-ai-gateway-infra(b4206617-2601-491e-a274-cb89b62cc103)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" podUID="b4206617-2601-491e-a274-cb89b62cc103"
Oct 30 13:45:20.243237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.243210    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" event={"ID":"81d67356-8625-4b93-88fe-b33f21dc9b8e","Type":"ContainerStarted","Data":"8d2fdd991b3145f2b59bc104b6abaf369afb1ba1f329ce8e3817f7903f3ea290"}
Oct 30 13:45:20.247564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.247269    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerStarted","Data":"fa236e0f88a3cad7c6873f6e47a87eef02cc487a8907a4ddfe93abb941e2c1b7"}
Oct 30 13:45:20.255715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.254834    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" event={"ID":"5714a3ae-1af2-43fd-824c-e6536556a50f","Type":"ContainerStarted","Data":"f789c36ab689922383328c8a5aac00014718e80f35fd604c48732244422cec33"}
Oct 30 13:45:20.263729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.257388    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd06531d-fb03-48a0-864d-b3e33da93784" containerID="07605e29ca04db74af2396bfe0b1e1b01d7040f3825b98698e13d70b8dd9dc78" exitCode=60
Oct 30 13:45:20.263729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.257475    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" event={"ID":"fd06531d-fb03-48a0-864d-b3e33da93784","Type":"ContainerDied","Data":"07605e29ca04db74af2396bfe0b1e1b01d7040f3825b98698e13d70b8dd9dc78"}
Oct 30 13:45:20.263729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.257913    4291 scope.go:122] "RemoveContainer" containerID="07605e29ca04db74af2396bfe0b1e1b01d7040f3825b98698e13d70b8dd9dc78"
Oct 30 13:45:20.263729 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:20.258249    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889465-mt5wq_odh-ai-gateway-infra(fd06531d-fb03-48a0-864d-b3e33da93784)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" podUID="fd06531d-fb03-48a0-864d-b3e33da93784"
Oct 30 13:45:20.263729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.262748    4291 scope.go:122] "RemoveContainer" containerID="c02e90e0267c4fb6b93722d0822b4ff512e87bb5e4eee144ee8151066807fecc"
Oct 30 13:45:20.268963 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.268930    4291 generic.go:356] "Generic (PLEG): container finished" podID="998880a0-7ee4-452c-89ca-d2cc33a61c7a" containerID="c2d9f0634981e4d6f9221f4822a62628aabfea059c212d4e5d33090ab73c05a8" exitCode=0
Oct 30 13:45:20.269103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.269019    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4" event={"ID":"998880a0-7ee4-452c-89ca-d2cc33a61c7a","Type":"ContainerDied","Data":"c2d9f0634981e4d6f9221f4822a62628aabfea059c212d4e5d33090ab73c05a8"}
Oct 30 13:45:20.271223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.271194    4291 generic.go:356] "Generic (PLEG): container finished" podID="80678d19-5845-487b-9329-577e2cf9738e" containerID="63a845b891e13d2138ff980a633b2a496992beecfa55e7d8f84ac571a259282e" exitCode=60
Oct 30 13:45:20.272057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.271257    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerDied","Data":"63a845b891e13d2138ff980a633b2a496992beecfa55e7d8f84ac571a259282e"}
Oct 30 13:45:20.272057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.271777    4291 scope.go:122] "RemoveContainer" containerID="63a845b891e13d2138ff980a633b2a496992beecfa55e7d8f84ac571a259282e"
Oct 30 13:45:20.272807 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:20.272772    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw_odh-ai-gateway-infra(80678d19-5845-487b-9329-577e2cf9738e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" podUID="80678d19-5845-487b-9329-577e2cf9738e"
Oct 30 13:45:20.300893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.300862    4291 scope.go:122] "RemoveContainer" containerID="38fc631231ebf3d4200e379029a9ec0d0d30145563c137c623d9176394a816a0"
Oct 30 13:45:20.317929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.317885    4291 scope.go:122] "RemoveContainer" containerID="5a006298f66fbbb47fb03d26a3a5e821f1243fe998d595d127fa72f0bd6516fc"
Oct 30 13:45:20.319624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.319603    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:45:20.335555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.335528    4291 scope.go:122] "RemoveContainer" containerID="a4975d30345c5db66c695eeacd7eee32e99d2cfd090cc152c1e2a1b37b4ae8a0"
Oct 30 13:45:20.369074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.369035    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"]
Oct 30 13:45:20.376218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.376182    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-routing-overlay-volume\") pod \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") "
Oct 30 13:45:20.376465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.376425    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-provider-credentials\") pod \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") "
Oct 30 13:45:20.376596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.376474    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-plugins-config-volume\") pod \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") "
Oct 30 13:45:20.376596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.376508    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-kube-api-access-cgxt6\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-kube-api-access-cgxt6\") pod \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\" (UID: \"998880a0-7ee4-452c-89ca-d2cc33a61c7a\") "
Oct 30 13:45:20.379678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.378224    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-routing-overlay-volume" pod "998880a0-7ee4-452c-89ca-d2cc33a61c7a" (UID: "998880a0-7ee4-452c-89ca-d2cc33a61c7a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:20.379678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.378942    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.379678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.379592    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-plugins-config-volume" pod "998880a0-7ee4-452c-89ca-d2cc33a61c7a" (UID: "998880a0-7ee4-452c-89ca-d2cc33a61c7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:20.385457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.385426    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"]
Oct 30 13:45:20.430776 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.430741    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"]
Oct 30 13:45:20.447432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.447395    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"]
Oct 30 13:45:20.451165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.450383    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-provider-credentials" pod "998880a0-7ee4-452c-89ca-d2cc33a61c7a" (UID: "998880a0-7ee4-452c-89ca-d2cc33a61c7a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:20.451165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.450430    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-kube-api-access-cgxt6" pod "998880a0-7ee4-452c-89ca-d2cc33a61c7a" (UID: "998880a0-7ee4-452c-89ca-d2cc33a61c7a"). InnerVolumeSpecName "kube-api-access-cgxt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:20.478657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.478612    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-52af49-29889465-lmbk4"]
Oct 30 13:45:20.481068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.481039    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.481068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.481068    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/998880a0-7ee4-452c-89ca-d2cc33a61c7a-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.481300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.481083    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxt6\" (UniqueName: \"kubernetes.io/projected/998880a0-7ee4-452c-89ca-d2cc33a61c7a-kube-api-access-cgxt6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.572969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.572911    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"]
Oct 30 13:45:20.580098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.579930    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889465-w8gpz"]
Oct 30 13:45:20.585973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.585710    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"]
Oct 30 13:45:20.714221 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.714191    4291 scope.go:122] "RemoveContainer" containerID="81353d5f80abbbaf30cb3594e72d600e81db86790df954bb4a91044a36becc58"
Oct 30 13:45:20.720540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.720515    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:20.732129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.731939    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:20.785596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.785559    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-kube-api-access-bsrpp\" (UniqueName: \"kubernetes.io/projected/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-kube-api-access-bsrpp\") pod \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") "
Oct 30 13:45:20.785802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.785700    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-maas-api-service-ca\") pod \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\" (UID: \"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b\") "
Oct 30 13:45:20.785982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.785966    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/696f6bbf-ec44-4546-8d71-29bd05cf026e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696f6bbf-ec44-4546-8d71-29bd05cf026e-maas-api-service-ca\") pod \"696f6bbf-ec44-4546-8d71-29bd05cf026e\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") "
Oct 30 13:45:20.786051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.786011    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/696f6bbf-ec44-4546-8d71-29bd05cf026e-kube-api-access-8f8wd\" (UniqueName: \"kubernetes.io/projected/696f6bbf-ec44-4546-8d71-29bd05cf026e-kube-api-access-8f8wd\") pod \"696f6bbf-ec44-4546-8d71-29bd05cf026e\" (UID: \"696f6bbf-ec44-4546-8d71-29bd05cf026e\") "
Oct 30 13:45:20.786106 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.786047    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-maas-api-service-ca" pod "bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" (UID: "bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:20.786375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.786340    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/696f6bbf-ec44-4546-8d71-29bd05cf026e-maas-api-service-ca" pod "696f6bbf-ec44-4546-8d71-29bd05cf026e" (UID: "696f6bbf-ec44-4546-8d71-29bd05cf026e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:20.786620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.786593    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/696f6bbf-ec44-4546-8d71-29bd05cf026e-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.786620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.786619    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.788126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.788104    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-kube-api-access-bsrpp" pod "bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" (UID: "bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b"). InnerVolumeSpecName "kube-api-access-bsrpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:20.788246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.788228    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/696f6bbf-ec44-4546-8d71-29bd05cf026e-kube-api-access-8f8wd" pod "696f6bbf-ec44-4546-8d71-29bd05cf026e" (UID: "696f6bbf-ec44-4546-8d71-29bd05cf026e"). InnerVolumeSpecName "kube-api-access-8f8wd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:20.871878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.871846    4291 scope.go:122] "RemoveContainer" containerID="5c337e9ffacdbceadec1c617680a9351ae08d78c8e67a0b2638d755c6b6ee3f1"
Oct 30 13:45:20.872078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.872007    4291 scope.go:122] "RemoveContainer" containerID="cec501c27d2b46a74e7139498dc5dfc35f36ea388ee388a4581549fe08f8e84f"
Oct 30 13:45:20.888062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.888020    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f8wd\" (UniqueName: \"kubernetes.io/projected/696f6bbf-ec44-4546-8d71-29bd05cf026e-kube-api-access-8f8wd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:20.888222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:20.888095    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bsrpp\" (UniqueName: \"kubernetes.io/projected/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b-kube-api-access-bsrpp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.284251 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.284216    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerStarted","Data":"650cabe2388bf69f788e65de2215304d5cd877f5fc27bf9082fae9b87b67b656"}
Oct 30 13:45:21.287134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.287101    4291 generic.go:356] "Generic (PLEG): container finished" podID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerID="2f791c0370295c01934b102104f5f3256df36e94d7f1c140f0bc9aaa95f3346f" exitCode=60
Oct 30 13:45:21.287271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.287143    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" event={"ID":"73b26525-0a60-471e-8fe4-2f5a9696b700","Type":"ContainerDied","Data":"2f791c0370295c01934b102104f5f3256df36e94d7f1c140f0bc9aaa95f3346f"}
Oct 30 13:45:21.287271 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.287211    4291 scope.go:122] "RemoveContainer" containerID="572cd39f2ba2765522583813824495dbf38742c08ac64459803be71ae4f06038"
Oct 30 13:45:21.287674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.287584    4291 scope.go:122] "RemoveContainer" containerID="2f791c0370295c01934b102104f5f3256df36e94d7f1c140f0bc9aaa95f3346f"
Oct 30 13:45:21.288233 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:21.288196    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9_odh-ai-gateway-infra(73b26525-0a60-471e-8fe4-2f5a9696b700)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700"
Oct 30 13:45:21.290680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.290626    4291 generic.go:356] "Generic (PLEG): container finished" podID="46831069-b252-4c2d-87d4-834789ce4d6e" containerID="655a7c61e5a0908d06e67a5acfc6a1f02bd0e7cd73734f1904e170e8f2ab67e8" exitCode=60
Oct 30 13:45:21.290777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.290687    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" event={"ID":"46831069-b252-4c2d-87d4-834789ce4d6e","Type":"ContainerDied","Data":"655a7c61e5a0908d06e67a5acfc6a1f02bd0e7cd73734f1904e170e8f2ab67e8"}
Oct 30 13:45:21.291045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.291030    4291 scope.go:122] "RemoveContainer" containerID="655a7c61e5a0908d06e67a5acfc6a1f02bd0e7cd73734f1904e170e8f2ab67e8"
Oct 30 13:45:21.291330 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:21.291308    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499_odh-ai-gateway-infra(46831069-b252-4c2d-87d4-834789ce4d6e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" podUID="46831069-b252-4c2d-87d4-834789ce4d6e"
Oct 30 13:45:21.293248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.293223    4291 generic.go:356] "Generic (PLEG): container finished" podID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerID="4f4b283a89b19149e770f4508d08f7bcbaaf87207b1153bce369e2bfaba24e67" exitCode=60
Oct 30 13:45:21.293354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.293316    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerDied","Data":"4f4b283a89b19149e770f4508d08f7bcbaaf87207b1153bce369e2bfaba24e67"}
Oct 30 13:45:21.293614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.293598    4291 scope.go:122] "RemoveContainer" containerID="4f4b283a89b19149e770f4508d08f7bcbaaf87207b1153bce369e2bfaba24e67"
Oct 30 13:45:21.293889 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:21.293859    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848_odh-ai-gateway-infra(5aaca5e6-383c-46b3-84b5-b5aa882e37c0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0"
Oct 30 13:45:21.297945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.297920    4291 generic.go:356] "Generic (PLEG): container finished" podID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerID="8d2fdd991b3145f2b59bc104b6abaf369afb1ba1f329ce8e3817f7903f3ea290" exitCode=60
Oct 30 13:45:21.298079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.297952    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" event={"ID":"81d67356-8625-4b93-88fe-b33f21dc9b8e","Type":"ContainerDied","Data":"8d2fdd991b3145f2b59bc104b6abaf369afb1ba1f329ce8e3817f7903f3ea290"}
Oct 30 13:45:21.298448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.298428    4291 scope.go:122] "RemoveContainer" containerID="8d2fdd991b3145f2b59bc104b6abaf369afb1ba1f329ce8e3817f7903f3ea290"
Oct 30 13:45:21.298867 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:21.298836    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf_odh-ai-gateway-infra(81d67356-8625-4b93-88fe-b33f21dc9b8e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e"
Oct 30 13:45:21.300426 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.300403    4291 generic.go:356] "Generic (PLEG): container finished" podID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerID="fa236e0f88a3cad7c6873f6e47a87eef02cc487a8907a4ddfe93abb941e2c1b7" exitCode=60
Oct 30 13:45:21.300528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.300491    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerDied","Data":"fa236e0f88a3cad7c6873f6e47a87eef02cc487a8907a4ddfe93abb941e2c1b7"}
Oct 30 13:45:21.301126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.301010    4291 scope.go:122] "RemoveContainer" containerID="fa236e0f88a3cad7c6873f6e47a87eef02cc487a8907a4ddfe93abb941e2c1b7"
Oct 30 13:45:21.301393 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:21.301368    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29_odh-ai-gateway-infra(ba738823-8bdf-4597-8eed-7e3a3fb027f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2"
Oct 30 13:45:21.306680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.302166    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"
Oct 30 13:45:21.306680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.302175    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx" event={"ID":"bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b","Type":"ContainerDied","Data":"09fbcf628a9a55f77eb78d188e1e89d853a08bade7c54905106098a1632ff175"}
Oct 30 13:45:21.306680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.304503    4291 generic.go:356] "Generic (PLEG): container finished" podID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerID="f789c36ab689922383328c8a5aac00014718e80f35fd604c48732244422cec33" exitCode=60
Oct 30 13:45:21.306680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.304578    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" event={"ID":"5714a3ae-1af2-43fd-824c-e6536556a50f","Type":"ContainerDied","Data":"f789c36ab689922383328c8a5aac00014718e80f35fd604c48732244422cec33"}
Oct 30 13:45:21.306680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.304815    4291 scope.go:122] "RemoveContainer" containerID="f6b3d54c540d40dcc27d6feda01470da8899ed6b94867b46758d29cd762d9648"
Oct 30 13:45:21.306680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.305194    4291 scope.go:122] "RemoveContainer" containerID="f789c36ab689922383328c8a5aac00014718e80f35fd604c48732244422cec33"
Oct 30 13:45:21.306680 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:21.305492    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d_odh-ai-gateway-infra(5714a3ae-1af2-43fd-824c-e6536556a50f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f"
Oct 30 13:45:21.312920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.312895    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"]
Oct 30 13:45:21.313926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.313597    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"
Oct 30 13:45:21.313926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.313592    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz" event={"ID":"696f6bbf-ec44-4546-8d71-29bd05cf026e","Type":"ContainerDied","Data":"d8d8d8d325211367ec62d3fbacf25cc0d4f60ca2e80fdff40bee84f93cbaad07"}
Oct 30 13:45:21.315622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.315595    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"
Oct 30 13:45:21.315922 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.315599    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4" event={"ID":"998880a0-7ee4-452c-89ca-d2cc33a61c7a","Type":"ContainerDied","Data":"f099afb158ba9ba2dff8ed16515d5e4191f31cebb11ded173bf50ae72eb3cddc"}
Oct 30 13:45:21.325720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.325694    4291 scope.go:122] "RemoveContainer" containerID="e2105aa159c9dcfc6dbc92aa73422e9e61f37f07a853d1e6bfc3f3f4692365ff"
Oct 30 13:45:21.392062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.392026    4291 scope.go:122] "RemoveContainer" containerID="2d8bdf1ff9993ab8547d0efd043c268f9e872142e22708984c00b5bce427009a"
Oct 30 13:45:21.442013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.441987    4291 scope.go:122] "RemoveContainer" containerID="a0f5c54a70d91b37e8816cc4ea33520ccca9837ee24a7dd3f94dd87baed24392"
Oct 30 13:45:21.466627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.466554    4291 scope.go:122] "RemoveContainer" containerID="c0089aca5ad5bf7c1d40171def1a6a4fbe60a8c62c22b2d0cd5338dea57b2691"
Oct 30 13:45:21.484456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.484423    4291 scope.go:122] "RemoveContainer" containerID="7efe4e17226279762c8f656d1768a56e5de80530f11d8e52e75cece08a66a8fb"
Oct 30 13:45:21.542946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.542901    4291 scope.go:122] "RemoveContainer" containerID="cb837c67c229077fa5e60d2e93f511d4d55aab0141ad0c1d8c10f572e3982b21"
Oct 30 13:45:21.548839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.548786    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:21.563063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.562611    4291 scope.go:122] "RemoveContainer" containerID="c2d9f0634981e4d6f9221f4822a62628aabfea059c212d4e5d33090ab73c05a8"
Oct 30 13:45:21.579574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.579545    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"]
Oct 30 13:45:21.585310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.585284    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-29a3f3-29889465-vhtpx"]
Oct 30 13:45:21.616730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.616697    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"]
Oct 30 13:45:21.617608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.617577    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:21.619739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.619716    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:21.623217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.623194    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-84687zlk4"]
Oct 30 13:45:21.630098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.630074    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75451d0a-8ddb-411c-98c7-b4fa2101d048-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75451d0a-8ddb-411c-98c7-b4fa2101d048-maas-api-service-ca\") pod \"75451d0a-8ddb-411c-98c7-b4fa2101d048\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") "
Oct 30 13:45:21.630351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.630328    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75451d0a-8ddb-411c-98c7-b4fa2101d048-kube-api-access-sn8hk\" (UniqueName: \"kubernetes.io/projected/75451d0a-8ddb-411c-98c7-b4fa2101d048-kube-api-access-sn8hk\") pod \"75451d0a-8ddb-411c-98c7-b4fa2101d048\" (UID: \"75451d0a-8ddb-411c-98c7-b4fa2101d048\") "
Oct 30 13:45:21.630479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.630461    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75451d0a-8ddb-411c-98c7-b4fa2101d048-maas-api-service-ca" pod "75451d0a-8ddb-411c-98c7-b4fa2101d048" (UID: "75451d0a-8ddb-411c-98c7-b4fa2101d048"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:21.630885 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.630867    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75451d0a-8ddb-411c-98c7-b4fa2101d048-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.632541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.632520    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75451d0a-8ddb-411c-98c7-b4fa2101d048-kube-api-access-sn8hk" pod "75451d0a-8ddb-411c-98c7-b4fa2101d048" (UID: "75451d0a-8ddb-411c-98c7-b4fa2101d048"). InnerVolumeSpecName "kube-api-access-sn8hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:21.638027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.638005    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"]
Oct 30 13:45:21.678079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.678042    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"]
Oct 30 13:45:21.715829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.715775    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"]
Oct 30 13:45:21.731522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.731478    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80678d19-5845-487b-9329-577e2cf9738e-kube-api-access-tb4hk\" (UniqueName: \"kubernetes.io/projected/80678d19-5845-487b-9329-577e2cf9738e-kube-api-access-tb4hk\") pod \"80678d19-5845-487b-9329-577e2cf9738e\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") "
Oct 30 13:45:21.731770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.731574    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd06531d-fb03-48a0-864d-b3e33da93784-kube-api-access-lsxtk\" (UniqueName: \"kubernetes.io/projected/fd06531d-fb03-48a0-864d-b3e33da93784-kube-api-access-lsxtk\") pod \"fd06531d-fb03-48a0-864d-b3e33da93784\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") "
Oct 30 13:45:21.731770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.731626    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd06531d-fb03-48a0-864d-b3e33da93784-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd06531d-fb03-48a0-864d-b3e33da93784-maas-api-service-ca\") pod \"fd06531d-fb03-48a0-864d-b3e33da93784\" (UID: \"fd06531d-fb03-48a0-864d-b3e33da93784\") "
Oct 30 13:45:21.731770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.731729    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80678d19-5845-487b-9329-577e2cf9738e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80678d19-5845-487b-9329-577e2cf9738e-maas-api-service-ca\") pod \"80678d19-5845-487b-9329-577e2cf9738e\" (UID: \"80678d19-5845-487b-9329-577e2cf9738e\") "
Oct 30 13:45:21.732046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.732019    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd06531d-fb03-48a0-864d-b3e33da93784-maas-api-service-ca" pod "fd06531d-fb03-48a0-864d-b3e33da93784" (UID: "fd06531d-fb03-48a0-864d-b3e33da93784"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:21.732100 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.732086    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80678d19-5845-487b-9329-577e2cf9738e-maas-api-service-ca" pod "80678d19-5845-487b-9329-577e2cf9738e" (UID: "80678d19-5845-487b-9329-577e2cf9738e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:21.732281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.732257    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd06531d-fb03-48a0-864d-b3e33da93784-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.732281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.732282    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn8hk\" (UniqueName: \"kubernetes.io/projected/75451d0a-8ddb-411c-98c7-b4fa2101d048-kube-api-access-sn8hk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.732479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.732293    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80678d19-5845-487b-9329-577e2cf9738e-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.733829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.733804    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80678d19-5845-487b-9329-577e2cf9738e-kube-api-access-tb4hk" pod "80678d19-5845-487b-9329-577e2cf9738e" (UID: "80678d19-5845-487b-9329-577e2cf9738e"). InnerVolumeSpecName "kube-api-access-tb4hk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:21.733936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.733851    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd06531d-fb03-48a0-864d-b3e33da93784-kube-api-access-lsxtk" pod "fd06531d-fb03-48a0-864d-b3e33da93784" (UID: "fd06531d-fb03-48a0-864d-b3e33da93784"). InnerVolumeSpecName "kube-api-access-lsxtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:21.734995 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.734974    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"]
Oct 30 13:45:21.739336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.739313    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-3f24fd-29889465-pr7gz"]
Oct 30 13:45:21.796730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.796690    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"]
Oct 30 13:45:21.833950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.833860    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb4hk\" (UniqueName: \"kubernetes.io/projected/80678d19-5845-487b-9329-577e2cf9738e-kube-api-access-tb4hk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.833950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.833890    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsxtk\" (UniqueName: \"kubernetes.io/projected/fd06531d-fb03-48a0-864d-b3e33da93784-kube-api-access-lsxtk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:21.839611 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.839578    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"]
Oct 30 13:45:21.882772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.882737    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" path="/var/lib/kubelet/pods/392207f2-fca4-4e13-80b4-218ff561c46f/volumes"
Oct 30 13:45:21.883349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.883330    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" path="/var/lib/kubelet/pods/5ec89a7d-eb27-48de-a793-86a7b0d5389b/volumes"
Oct 30 13:45:21.883795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.883780    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" path="/var/lib/kubelet/pods/696f6bbf-ec44-4546-8d71-29bd05cf026e/volumes"
Oct 30 13:45:21.884529 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.884512    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="998880a0-7ee4-452c-89ca-d2cc33a61c7a" path="/var/lib/kubelet/pods/998880a0-7ee4-452c-89ca-d2cc33a61c7a/volumes"
Oct 30 13:45:21.884938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:21.884924    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" path="/var/lib/kubelet/pods/bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b/volumes"
Oct 30 13:45:22.044014 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.043975    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"]
Oct 30 13:45:22.212699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.212671    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:22.334165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.334140    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"
Oct 30 13:45:22.334585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.334159    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq" event={"ID":"fd06531d-fb03-48a0-864d-b3e33da93784","Type":"ContainerDied","Data":"d68040ae3106f8a320955b286e4dd358b4e74ab14cd350f8a304972f660570bc"}
Oct 30 13:45:22.334585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.334222    4291 scope.go:122] "RemoveContainer" containerID="07605e29ca04db74af2396bfe0b1e1b01d7040f3825b98698e13d70b8dd9dc78"
Oct 30 13:45:22.341523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.340655    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4206617-2601-491e-a274-cb89b62cc103-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4206617-2601-491e-a274-cb89b62cc103-maas-api-service-ca\") pod \"b4206617-2601-491e-a274-cb89b62cc103\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") "
Oct 30 13:45:22.341523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.340723    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4206617-2601-491e-a274-cb89b62cc103-kube-api-access-q5gbv\" (UniqueName: \"kubernetes.io/projected/b4206617-2601-491e-a274-cb89b62cc103-kube-api-access-q5gbv\") pod \"b4206617-2601-491e-a274-cb89b62cc103\" (UID: \"b4206617-2601-491e-a274-cb89b62cc103\") "
Oct 30 13:45:22.341523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.341046    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw" event={"ID":"80678d19-5845-487b-9329-577e2cf9738e","Type":"ContainerDied","Data":"74401d8ad9997aa06fb40ef5cb6abde513ded966dbdb112bf99efa4ddf94a447"}
Oct 30 13:45:22.341523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.341160    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"
Oct 30 13:45:22.342188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.341998    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4206617-2601-491e-a274-cb89b62cc103-maas-api-service-ca" pod "b4206617-2601-491e-a274-cb89b62cc103" (UID: "b4206617-2601-491e-a274-cb89b62cc103"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:22.342968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.342930    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4206617-2601-491e-a274-cb89b62cc103-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.344882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.344766    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4206617-2601-491e-a274-cb89b62cc103-kube-api-access-q5gbv" pod "b4206617-2601-491e-a274-cb89b62cc103" (UID: "b4206617-2601-491e-a274-cb89b62cc103"). InnerVolumeSpecName "kube-api-access-q5gbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:22.344983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.344961    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"
Oct 30 13:45:22.345831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.345733    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr" event={"ID":"75451d0a-8ddb-411c-98c7-b4fa2101d048","Type":"ContainerDied","Data":"b64773558eb77896822e3584fa5f61259b1cefee228143f2d7e2feef916cdc40"}
Oct 30 13:45:22.348695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.347924    4291 generic.go:356] "Generic (PLEG): container finished" podID="7253bff1-99f9-4619-9239-3a76834a05dc" containerID="650cabe2388bf69f788e65de2215304d5cd877f5fc27bf9082fae9b87b67b656" exitCode=60
Oct 30 13:45:22.348695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.348064    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerDied","Data":"650cabe2388bf69f788e65de2215304d5cd877f5fc27bf9082fae9b87b67b656"}
Oct 30 13:45:22.348695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.348480    4291 scope.go:122] "RemoveContainer" containerID="650cabe2388bf69f788e65de2215304d5cd877f5fc27bf9082fae9b87b67b656"
Oct 30 13:45:22.349013 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:22.348811    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc_odh-ai-gateway-infra(7253bff1-99f9-4619-9239-3a76834a05dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" podUID="7253bff1-99f9-4619-9239-3a76834a05dc"
Oct 30 13:45:22.359063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.357828    4291 scope.go:122] "RemoveContainer" containerID="63a845b891e13d2138ff980a633b2a496992beecfa55e7d8f84ac571a259282e"
Oct 30 13:45:22.364363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.364283    4291 generic.go:356] "Generic (PLEG): container finished" podID="32860ef1-4fcb-45f2-8297-f3330e738442" containerID="5d851a8680b31629cc1a857f3fdb6e1be63bf335bc5b861cf7d5a051cba9d31c" exitCode=60
Oct 30 13:45:22.364479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.364430    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" event={"ID":"32860ef1-4fcb-45f2-8297-f3330e738442","Type":"ContainerDied","Data":"5d851a8680b31629cc1a857f3fdb6e1be63bf335bc5b861cf7d5a051cba9d31c"}
Oct 30 13:45:22.365200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.365078    4291 scope.go:122] "RemoveContainer" containerID="5d851a8680b31629cc1a857f3fdb6e1be63bf335bc5b861cf7d5a051cba9d31c"
Oct 30 13:45:22.366096 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:22.365748    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq_odh-ai-gateway-infra(32860ef1-4fcb-45f2-8297-f3330e738442)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" podUID="32860ef1-4fcb-45f2-8297-f3330e738442"
Oct 30 13:45:22.383690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.379083    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn" event={"ID":"b4206617-2601-491e-a274-cb89b62cc103","Type":"ContainerDied","Data":"b28934a959ebd5013b86ee703ccf37ddf5f406c5e5a0a29be25ef0d7d70a285c"}
Oct 30 13:45:22.383690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.379202    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"
Oct 30 13:45:22.526096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.513918    4291 scope.go:122] "RemoveContainer" containerID="fea266ac2b58a80c350da01e32d48d1c96990490597ca6d882b3bb2f236dde25"
Oct 30 13:45:22.526096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.520139    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5gbv\" (UniqueName: \"kubernetes.io/projected/b4206617-2601-491e-a274-cb89b62cc103-kube-api-access-q5gbv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.557360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.557296    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"]
Oct 30 13:45:22.580224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.580198    4291 scope.go:122] "RemoveContainer" containerID="cec501c27d2b46a74e7139498dc5dfc35f36ea388ee388a4581549fe08f8e84f"
Oct 30 13:45:22.608093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.607905    4291 scope.go:122] "RemoveContainer" containerID="5c337e9ffacdbceadec1c617680a9351ae08d78c8e67a0b2638d755c6b6ee3f1"
Oct 30 13:45:22.629463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.629433    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"]
Oct 30 13:45:22.635670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.635626    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889465-ckrcr"]
Oct 30 13:45:22.665757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.665721    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"]
Oct 30 13:45:22.673446 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.673415    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889465-xj2qw"]
Oct 30 13:45:22.678117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.678089    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:22.679399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.679380    4291 scope.go:122] "RemoveContainer" containerID="1da92cc55d3edb93f9413c4ba2671208e15dac862b42f3d8a7b4bfd74d27eb3f"
Oct 30 13:45:22.785945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.785909    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:22.825547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.825515    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81d67356-8625-4b93-88fe-b33f21dc9b8e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d67356-8625-4b93-88fe-b33f21dc9b8e-maas-api-service-ca\") pod \"81d67356-8625-4b93-88fe-b33f21dc9b8e\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") "
Oct 30 13:45:22.825750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.825622    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81d67356-8625-4b93-88fe-b33f21dc9b8e-kube-api-access-r6z8j\" (UniqueName: \"kubernetes.io/projected/81d67356-8625-4b93-88fe-b33f21dc9b8e-kube-api-access-r6z8j\") pod \"81d67356-8625-4b93-88fe-b33f21dc9b8e\" (UID: \"81d67356-8625-4b93-88fe-b33f21dc9b8e\") "
Oct 30 13:45:22.826056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.826028    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81d67356-8625-4b93-88fe-b33f21dc9b8e-maas-api-service-ca" pod "81d67356-8625-4b93-88fe-b33f21dc9b8e" (UID: "81d67356-8625-4b93-88fe-b33f21dc9b8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:22.826589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.826480    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/81d67356-8625-4b93-88fe-b33f21dc9b8e-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.829156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.829124    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81d67356-8625-4b93-88fe-b33f21dc9b8e-kube-api-access-r6z8j" pod "81d67356-8625-4b93-88fe-b33f21dc9b8e" (UID: "81d67356-8625-4b93-88fe-b33f21dc9b8e"). InnerVolumeSpecName "kube-api-access-r6z8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:22.829725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.829704    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:22.851316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.851283    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:22.865081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.865039    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:22.867702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.867676    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:22.884348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.884264    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"]
Oct 30 13:45:22.894496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.894464    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889465-mt5wq"]
Oct 30 13:45:22.926978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.926938    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b26525-0a60-471e-8fe4-2f5a9696b700-kube-api-access-zgf2z\" (UniqueName: \"kubernetes.io/projected/73b26525-0a60-471e-8fe4-2f5a9696b700-kube-api-access-zgf2z\") pod \"73b26525-0a60-471e-8fe4-2f5a9696b700\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") "
Oct 30 13:45:22.927225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.927204    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5714a3ae-1af2-43fd-824c-e6536556a50f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714a3ae-1af2-43fd-824c-e6536556a50f-maas-api-service-ca\") pod \"5714a3ae-1af2-43fd-824c-e6536556a50f\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") "
Oct 30 13:45:22.927311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.927294    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5714a3ae-1af2-43fd-824c-e6536556a50f-kube-api-access-tw6z8\" (UniqueName: \"kubernetes.io/projected/5714a3ae-1af2-43fd-824c-e6536556a50f-kube-api-access-tw6z8\") pod \"5714a3ae-1af2-43fd-824c-e6536556a50f\" (UID: \"5714a3ae-1af2-43fd-824c-e6536556a50f\") "
Oct 30 13:45:22.927372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.927345    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73b26525-0a60-471e-8fe4-2f5a9696b700-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b26525-0a60-471e-8fe4-2f5a9696b700-maas-api-service-ca\") pod \"73b26525-0a60-471e-8fe4-2f5a9696b700\" (UID: \"73b26525-0a60-471e-8fe4-2f5a9696b700\") "
Oct 30 13:45:22.927716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.927669    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5714a3ae-1af2-43fd-824c-e6536556a50f-maas-api-service-ca" pod "5714a3ae-1af2-43fd-824c-e6536556a50f" (UID: "5714a3ae-1af2-43fd-824c-e6536556a50f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:22.927841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.927743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73b26525-0a60-471e-8fe4-2f5a9696b700-maas-api-service-ca" pod "73b26525-0a60-471e-8fe4-2f5a9696b700" (UID: "73b26525-0a60-471e-8fe4-2f5a9696b700"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:22.928056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.927992    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73b26525-0a60-471e-8fe4-2f5a9696b700-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.928056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.928011    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6z8j\" (UniqueName: \"kubernetes.io/projected/81d67356-8625-4b93-88fe-b33f21dc9b8e-kube-api-access-r6z8j\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.928056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.928021    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5714a3ae-1af2-43fd-824c-e6536556a50f-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:22.929385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.929356    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b26525-0a60-471e-8fe4-2f5a9696b700-kube-api-access-zgf2z" pod "73b26525-0a60-471e-8fe4-2f5a9696b700" (UID: "73b26525-0a60-471e-8fe4-2f5a9696b700"). InnerVolumeSpecName "kube-api-access-zgf2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:22.929538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:22.929428    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5714a3ae-1af2-43fd-824c-e6536556a50f-kube-api-access-tw6z8" pod "5714a3ae-1af2-43fd-824c-e6536556a50f" (UID: "5714a3ae-1af2-43fd-824c-e6536556a50f"). InnerVolumeSpecName "kube-api-access-tw6z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.028574 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.028522    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba738823-8bdf-4597-8eed-7e3a3fb027f2-kube-api-access-rt8tf\" (UniqueName: \"kubernetes.io/projected/ba738823-8bdf-4597-8eed-7e3a3fb027f2-kube-api-access-rt8tf\") pod \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") "
Oct 30 13:45:23.028815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.028596    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-maas-api-service-ca\") pod \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") "
Oct 30 13:45:23.028815 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.028762    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-kube-api-access-77k4h\" (UniqueName: \"kubernetes.io/projected/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-kube-api-access-77k4h\") pod \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\" (UID: \"5aaca5e6-383c-46b3-84b5-b5aa882e37c0\") "
Oct 30 13:45:23.029002 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.028876    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46831069-b252-4c2d-87d4-834789ce4d6e-kube-api-access-rcp6p\" (UniqueName: \"kubernetes.io/projected/46831069-b252-4c2d-87d4-834789ce4d6e-kube-api-access-rcp6p\") pod \"46831069-b252-4c2d-87d4-834789ce4d6e\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") "
Oct 30 13:45:23.029078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029027    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba738823-8bdf-4597-8eed-7e3a3fb027f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba738823-8bdf-4597-8eed-7e3a3fb027f2-maas-api-service-ca\") pod \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\" (UID: \"ba738823-8bdf-4597-8eed-7e3a3fb027f2\") "
Oct 30 13:45:23.029078 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029062    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46831069-b252-4c2d-87d4-834789ce4d6e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46831069-b252-4c2d-87d4-834789ce4d6e-maas-api-service-ca\") pod \"46831069-b252-4c2d-87d4-834789ce4d6e\" (UID: \"46831069-b252-4c2d-87d4-834789ce4d6e\") "
Oct 30 13:45:23.029281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029250    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-maas-api-service-ca" pod "5aaca5e6-383c-46b3-84b5-b5aa882e37c0" (UID: "5aaca5e6-383c-46b3-84b5-b5aa882e37c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:23.030548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029666    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46831069-b252-4c2d-87d4-834789ce4d6e-maas-api-service-ca" pod "46831069-b252-4c2d-87d4-834789ce4d6e" (UID: "46831069-b252-4c2d-87d4-834789ce4d6e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:23.030548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029761    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba738823-8bdf-4597-8eed-7e3a3fb027f2-maas-api-service-ca" pod "ba738823-8bdf-4597-8eed-7e3a3fb027f2" (UID: "ba738823-8bdf-4597-8eed-7e3a3fb027f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:23.030548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029936    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.030548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029956    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tw6z8\" (UniqueName: \"kubernetes.io/projected/5714a3ae-1af2-43fd-824c-e6536556a50f-kube-api-access-tw6z8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.030548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029970    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgf2z\" (UniqueName: \"kubernetes.io/projected/73b26525-0a60-471e-8fe4-2f5a9696b700-kube-api-access-zgf2z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.030548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029982    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba738823-8bdf-4597-8eed-7e3a3fb027f2-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.030548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.029995    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/46831069-b252-4c2d-87d4-834789ce4d6e-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.031366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.031333    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba738823-8bdf-4597-8eed-7e3a3fb027f2-kube-api-access-rt8tf" pod "ba738823-8bdf-4597-8eed-7e3a3fb027f2" (UID: "ba738823-8bdf-4597-8eed-7e3a3fb027f2"). InnerVolumeSpecName "kube-api-access-rt8tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.032063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.032025    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46831069-b252-4c2d-87d4-834789ce4d6e-kube-api-access-rcp6p" pod "46831069-b252-4c2d-87d4-834789ce4d6e" (UID: "46831069-b252-4c2d-87d4-834789ce4d6e"). InnerVolumeSpecName "kube-api-access-rcp6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.032158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.032133    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-kube-api-access-77k4h" pod "5aaca5e6-383c-46b3-84b5-b5aa882e37c0" (UID: "5aaca5e6-383c-46b3-84b5-b5aa882e37c0"). InnerVolumeSpecName "kube-api-access-77k4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.131709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.131673    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt8tf\" (UniqueName: \"kubernetes.io/projected/ba738823-8bdf-4597-8eed-7e3a3fb027f2-kube-api-access-rt8tf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.131709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.131707    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77k4h\" (UniqueName: \"kubernetes.io/projected/5aaca5e6-383c-46b3-84b5-b5aa882e37c0-kube-api-access-77k4h\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.131709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.131725    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcp6p\" (UniqueName: \"kubernetes.io/projected/46831069-b252-4c2d-87d4-834789ce4d6e-kube-api-access-rcp6p\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.239254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.239172    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"]
Oct 30 13:45:23.243587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.243563    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889465-vwdxn"]
Oct 30 13:45:23.387868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.387832    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"
Oct 30 13:45:23.388330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.387830    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d" event={"ID":"5714a3ae-1af2-43fd-824c-e6536556a50f","Type":"ContainerDied","Data":"3c23d390516d2f3d7fe4d2f11764565ac1df22f363b4c7c650d940edddac8a41"}
Oct 30 13:45:23.388330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.387971    4291 scope.go:122] "RemoveContainer" containerID="f789c36ab689922383328c8a5aac00014718e80f35fd604c48732244422cec33"
Oct 30 13:45:23.394312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.394287    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"
Oct 30 13:45:23.394312 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.394300    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9" event={"ID":"73b26525-0a60-471e-8fe4-2f5a9696b700","Type":"ContainerDied","Data":"0dc642e108e6ea764b53e50cf644847adea50d043cf5a3e8044747fcb49ee965"}
Oct 30 13:45:23.400724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.397865    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"
Oct 30 13:45:23.400724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.397886    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499" event={"ID":"46831069-b252-4c2d-87d4-834789ce4d6e","Type":"ContainerDied","Data":"0eda99b3eb877f61adc7db0bd89f03a71ec01ad8da9d6b0b7728921e7fceac42"}
Oct 30 13:45:23.404202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.402784    4291 scope.go:122] "RemoveContainer" containerID="2f791c0370295c01934b102104f5f3256df36e94d7f1c140f0bc9aaa95f3346f"
Oct 30 13:45:23.405117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.404803    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848" event={"ID":"5aaca5e6-383c-46b3-84b5-b5aa882e37c0","Type":"ContainerDied","Data":"2506af3636247ca17b22b2f092723082e4e6c7d51fe2f3256a8609e30483d9b1"}
Oct 30 13:45:23.405117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.404864    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"
Oct 30 13:45:23.410686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.408827    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"
Oct 30 13:45:23.410686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.408857    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf" event={"ID":"81d67356-8625-4b93-88fe-b33f21dc9b8e","Type":"ContainerDied","Data":"0b135215d671e1e231395f17c01a21f88991a6c016fd66584e3e5eebad029890"}
Oct 30 13:45:23.418546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.418505    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29" event={"ID":"ba738823-8bdf-4597-8eed-7e3a3fb027f2","Type":"ContainerDied","Data":"d0b22c0a6dd4a56eb1c0eaa05c5ef5fb14907780ff789e18fc4697039afcbc4c"}
Oct 30 13:45:23.418695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.418551    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"
Oct 30 13:45:23.444124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.444104    4291 scope.go:122] "RemoveContainer" containerID="655a7c61e5a0908d06e67a5acfc6a1f02bd0e7cd73734f1904e170e8f2ab67e8"
Oct 30 13:45:23.457871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.457838    4291 scope.go:122] "RemoveContainer" containerID="4f4b283a89b19149e770f4508d08f7bcbaaf87207b1153bce369e2bfaba24e67"
Oct 30 13:45:23.482151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.482121    4291 scope.go:122] "RemoveContainer" containerID="8d2fdd991b3145f2b59bc104b6abaf369afb1ba1f329ce8e3817f7903f3ea290"
Oct 30 13:45:23.496237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.496211    4291 scope.go:122] "RemoveContainer" containerID="fa236e0f88a3cad7c6873f6e47a87eef02cc487a8907a4ddfe93abb941e2c1b7"
Oct 30 13:45:23.523772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.523740    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"]
Oct 30 13:45:23.531767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.531704    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-d2cecb-29889465-q7499"]
Oct 30 13:45:23.566795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.566750    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:23.568262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.568212    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"]
Oct 30 13:45:23.574105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.574042    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-ec41e5-29889465-dhf29"]
Oct 30 13:45:23.606143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.606110    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"]
Oct 30 13:45:23.613501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.613471    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889465-65848"]
Oct 30 13:45:23.647480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.647444    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"]
Oct 30 13:45:23.651687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.651656    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889465-f8n9d"]
Oct 30 13:45:23.682035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.682002    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"]
Oct 30 13:45:23.688591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.688559    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-98f1cc-29889465-bbpn9"]
Oct 30 13:45:23.738619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.738582    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"]
Oct 30 13:45:23.741796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.741762    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7253bff1-99f9-4619-9239-3a76834a05dc-kube-api-access-ppzhc\" (UniqueName: \"kubernetes.io/projected/7253bff1-99f9-4619-9239-3a76834a05dc-kube-api-access-ppzhc\") pod \"7253bff1-99f9-4619-9239-3a76834a05dc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") "
Oct 30 13:45:23.742084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.742060    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7253bff1-99f9-4619-9239-3a76834a05dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7253bff1-99f9-4619-9239-3a76834a05dc-maas-api-service-ca\") pod \"7253bff1-99f9-4619-9239-3a76834a05dc\" (UID: \"7253bff1-99f9-4619-9239-3a76834a05dc\") "
Oct 30 13:45:23.742576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.742547    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7253bff1-99f9-4619-9239-3a76834a05dc-maas-api-service-ca" pod "7253bff1-99f9-4619-9239-3a76834a05dc" (UID: "7253bff1-99f9-4619-9239-3a76834a05dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:23.742875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.742854    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7253bff1-99f9-4619-9239-3a76834a05dc-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.744388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.744358    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7253bff1-99f9-4619-9239-3a76834a05dc-kube-api-access-ppzhc" pod "7253bff1-99f9-4619-9239-3a76834a05dc" (UID: "7253bff1-99f9-4619-9239-3a76834a05dc"). InnerVolumeSpecName "kube-api-access-ppzhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:23.744854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.744836    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889465-zckzf"]
Oct 30 13:45:23.843858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.843822    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppzhc\" (UniqueName: \"kubernetes.io/projected/7253bff1-99f9-4619-9239-3a76834a05dc-kube-api-access-ppzhc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:23.881704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.881662    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" path="/var/lib/kubelet/pods/46831069-b252-4c2d-87d4-834789ce4d6e/volumes"
Oct 30 13:45:23.882164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.882142    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" path="/var/lib/kubelet/pods/5714a3ae-1af2-43fd-824c-e6536556a50f/volumes"
Oct 30 13:45:23.882552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.882536    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" path="/var/lib/kubelet/pods/5aaca5e6-383c-46b3-84b5-b5aa882e37c0/volumes"
Oct 30 13:45:23.883266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.883249    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" path="/var/lib/kubelet/pods/73b26525-0a60-471e-8fe4-2f5a9696b700/volumes"
Oct 30 13:45:23.883679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.883657    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" path="/var/lib/kubelet/pods/75451d0a-8ddb-411c-98c7-b4fa2101d048/volumes"
Oct 30 13:45:23.884075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.884061    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80678d19-5845-487b-9329-577e2cf9738e" path="/var/lib/kubelet/pods/80678d19-5845-487b-9329-577e2cf9738e/volumes"
Oct 30 13:45:23.884758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.884743    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" path="/var/lib/kubelet/pods/81d67356-8625-4b93-88fe-b33f21dc9b8e/volumes"
Oct 30 13:45:23.885143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.885125    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4206617-2601-491e-a274-cb89b62cc103" path="/var/lib/kubelet/pods/b4206617-2601-491e-a274-cb89b62cc103/volumes"
Oct 30 13:45:23.885570 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.885556    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" path="/var/lib/kubelet/pods/ba738823-8bdf-4597-8eed-7e3a3fb027f2/volumes"
Oct 30 13:45:23.886260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.886246    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" path="/var/lib/kubelet/pods/fd06531d-fb03-48a0-864d-b3e33da93784/volumes"
Oct 30 13:45:23.944464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:23.944432    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"]
Oct 30 13:45:24.094730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.094706    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:24.147959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.147927    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32860ef1-4fcb-45f2-8297-f3330e738442-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32860ef1-4fcb-45f2-8297-f3330e738442-maas-api-service-ca\") pod \"32860ef1-4fcb-45f2-8297-f3330e738442\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") "
Oct 30 13:45:24.148168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.147993    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32860ef1-4fcb-45f2-8297-f3330e738442-kube-api-access-njxl4\" (UniqueName: \"kubernetes.io/projected/32860ef1-4fcb-45f2-8297-f3330e738442-kube-api-access-njxl4\") pod \"32860ef1-4fcb-45f2-8297-f3330e738442\" (UID: \"32860ef1-4fcb-45f2-8297-f3330e738442\") "
Oct 30 13:45:24.148704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.148674    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32860ef1-4fcb-45f2-8297-f3330e738442-maas-api-service-ca" pod "32860ef1-4fcb-45f2-8297-f3330e738442" (UID: "32860ef1-4fcb-45f2-8297-f3330e738442"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:24.150299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.150270    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32860ef1-4fcb-45f2-8297-f3330e738442-kube-api-access-njxl4" pod "32860ef1-4fcb-45f2-8297-f3330e738442" (UID: "32860ef1-4fcb-45f2-8297-f3330e738442"). InnerVolumeSpecName "kube-api-access-njxl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:24.249373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.249333    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/32860ef1-4fcb-45f2-8297-f3330e738442-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:24.249373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.249366    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njxl4\" (UniqueName: \"kubernetes.io/projected/32860ef1-4fcb-45f2-8297-f3330e738442-kube-api-access-njxl4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:24.431351 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.431319    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc" event={"ID":"7253bff1-99f9-4619-9239-3a76834a05dc","Type":"ContainerDied","Data":"4ed0b0b974247a7acd6daa3f08f59fd7a05b07a0d39fddccbb30a532e37e89b7"}
Oct 30 13:45:24.431854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.431331    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"
Oct 30 13:45:24.431854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.431374    4291 scope.go:122] "RemoveContainer" containerID="650cabe2388bf69f788e65de2215304d5cd877f5fc27bf9082fae9b87b67b656"
Oct 30 13:45:24.433891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.433869    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"
Oct 30 13:45:24.434059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.433996    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq" event={"ID":"32860ef1-4fcb-45f2-8297-f3330e738442","Type":"ContainerDied","Data":"9992d7d6cd09087449a29600c382e8d5c603caddce15a457f184f54c175368a1"}
Oct 30 13:45:24.444139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.444117    4291 scope.go:122] "RemoveContainer" containerID="5d851a8680b31629cc1a857f3fdb6e1be63bf335bc5b861cf7d5a051cba9d31c"
Oct 30 13:45:24.483794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.483766    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"]
Oct 30 13:45:24.487993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.487969    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889465-kzhvc"]
Oct 30 13:45:24.519403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.519362    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"]
Oct 30 13:45:24.523990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:24.523954    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889465-lgpfq"]
Oct 30 13:45:25.882437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:25.882213    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" path="/var/lib/kubelet/pods/32860ef1-4fcb-45f2-8297-f3330e738442/volumes"
Oct 30 13:45:25.882891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:25.882839    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" path="/var/lib/kubelet/pods/7253bff1-99f9-4619-9239-3a76834a05dc/volumes"
Oct 30 13:45:38.212420 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:45:38.212377    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:45:39.765581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:39.765542    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:45:49.805980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.805943    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"]
Oct 30 13:45:49.806587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806571    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:45:49.806587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806587    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806596    4291 cpu_manager.go:414] "containerMap: removing container" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806602    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806613    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806617    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806625    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806629    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806654    4291 cpu_manager.go:414] "containerMap: removing container" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806660    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806671    4291 cpu_manager.go:414] "containerMap: removing container" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806679    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806685    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806690    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:45:49.806694 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806698    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806704    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806714    4291 cpu_manager.go:414] "containerMap: removing container" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806719    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806726    4291 cpu_manager.go:414] "containerMap: removing container" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806730    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806737    4291 cpu_manager.go:414] "containerMap: removing container" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806740    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806747    4291 cpu_manager.go:414] "containerMap: removing container" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806751    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806760    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806764    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806773    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806777    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806785    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806789    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806795    4291 cpu_manager.go:414] "containerMap: removing container" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806799    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806804    4291 cpu_manager.go:414] "containerMap: removing container" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806808    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806816    4291 cpu_manager.go:414] "containerMap: removing container" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806820    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806827    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806831    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806837    4291 cpu_manager.go:414] "containerMap: removing container" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806842    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806847    4291 cpu_manager.go:414] "containerMap: removing container" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806851    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806858    4291 cpu_manager.go:414] "containerMap: removing container" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806862    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806868    4291 cpu_manager.go:414] "containerMap: removing container" podUID="998880a0-7ee4-452c-89ca-d2cc33a61c7a" containerName="payload-processing"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806873    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="998880a0-7ee4-452c-89ca-d2cc33a61c7a" containerName="payload-processing"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806880    4291 cpu_manager.go:414] "containerMap: removing container" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806885    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806892    4291 cpu_manager.go:414] "containerMap: removing container" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806896    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806904    4291 cpu_manager.go:414] "containerMap: removing container" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806908    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806913    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806918    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806926    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806931    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806937    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806941    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806948    4291 cpu_manager.go:414] "containerMap: removing container" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806952    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806960    4291 cpu_manager.go:414] "containerMap: removing container" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806965    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806972    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806978    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806983    4291 cpu_manager.go:414] "containerMap: removing container" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806988    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806993    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.806997    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807005    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807009    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807018    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807022    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807028    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807032    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807038    4291 cpu_manager.go:414] "containerMap: removing container" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807042    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807050    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807054    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807065    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807069    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807082    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807086    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807094    4291 cpu_manager.go:414] "containerMap: removing container" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807098    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807105    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807111    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807117    4291 cpu_manager.go:414] "containerMap: removing container" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807122    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807129    4291 cpu_manager.go:414] "containerMap: removing container" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:45:49.807162 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807133    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807253    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807261    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807268    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807274    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807280    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807285    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807290    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807295    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807300    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807305    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807310    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807315    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807320    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807326    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807331    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807335    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807340    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807345    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807352    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807357    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807361    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807367    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807373    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807379    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807384    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807389    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807394    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b26525-0a60-471e-8fe4-2f5a9696b700" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807400    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807406    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="998880a0-7ee4-452c-89ca-d2cc33a61c7a" containerName="payload-processing"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807410    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807416    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807420    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd85a77b-3edb-41e8-a2c4-cca4c4b1ff5b" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807426    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807431    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807436    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807442    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807446    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807452    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807456    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:45:49.809557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.807462    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ec89a7d-eb27-48de-a793-86a7b0d5389b" containerName="cleanup"
Oct 30 13:45:49.811398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.811378    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.828790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.828752    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.828993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.828819    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhnsf\" (UniqueName: \"kubernetes.io/projected/5a37256d-b608-425e-9b04-0b0db90480b7-kube-api-access-dhnsf\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.828993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.828963    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.830826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.830794    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"]
Oct 30 13:45:49.837671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.837622    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:45:49.837671 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.837624    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:45:49.837931 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.837764    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:45:49.839107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.839081    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:45:49.930942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.930901    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.931507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.931214    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.931507 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.931291    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhnsf\" (UniqueName: \"kubernetes.io/projected/5a37256d-b608-425e-9b04-0b0db90480b7-kube-api-access-dhnsf\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.931743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.931724    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.933779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.933750    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:49.954986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:49.954954    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhnsf\" (UniqueName: \"kubernetes.io/projected/5a37256d-b608-425e-9b04-0b0db90480b7-kube-api-access-dhnsf\") pod \"maas-api-revoke-sub-d46f453e-rkp7r\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:50.134408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:50.134371    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:50.451057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:50.451020    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"]
Oct 30 13:45:50.451793 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:45:50.451761    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a37256d_b608_425e_9b04_0b0db90480b7.slice/crio-79feedc74fbffa94ed2508077210065baca7006ca05efca3a20a25fac755dcf0 WatchSource:0}: Error finding container 79feedc74fbffa94ed2508077210065baca7006ca05efca3a20a25fac755dcf0: Status 404 returned error can't find the container with id 79feedc74fbffa94ed2508077210065baca7006ca05efca3a20a25fac755dcf0
Oct 30 13:45:50.835970 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:50.835934    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r" event={"ID":"5a37256d-b608-425e-9b04-0b0db90480b7","Type":"ContainerStarted","Data":"79feedc74fbffa94ed2508077210065baca7006ca05efca3a20a25fac755dcf0"}
Oct 30 13:45:51.845230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:51.845190    4291 generic.go:356] "Generic (PLEG): container finished" podID="5a37256d-b608-425e-9b04-0b0db90480b7" containerID="a146065da0f37cf2ae18f9590bb0f8d13c32958a255c0e772dfa304de13ff95e" exitCode=0
Oct 30 13:45:51.845623 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:51.845273    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r" event={"ID":"5a37256d-b608-425e-9b04-0b0db90480b7","Type":"ContainerDied","Data":"a146065da0f37cf2ae18f9590bb0f8d13c32958a255c0e772dfa304de13ff95e"}
Oct 30 13:45:53.020254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.020219    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:53.077652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.076238    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-openshift-ca\") pod \"5a37256d-b608-425e-9b04-0b0db90480b7\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") "
Oct 30 13:45:53.077652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.076311    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a37256d-b608-425e-9b04-0b0db90480b7-kube-api-access-dhnsf\" (UniqueName: \"kubernetes.io/projected/5a37256d-b608-425e-9b04-0b0db90480b7-kube-api-access-dhnsf\") pod \"5a37256d-b608-425e-9b04-0b0db90480b7\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") "
Oct 30 13:45:53.077652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.076462    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-xks-ca\") pod \"5a37256d-b608-425e-9b04-0b0db90480b7\" (UID: \"5a37256d-b608-425e-9b04-0b0db90480b7\") "
Oct 30 13:45:53.077652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.076697    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-openshift-ca" pod "5a37256d-b608-425e-9b04-0b0db90480b7" (UID: "5a37256d-b608-425e-9b04-0b0db90480b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:45:53.077652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.077237    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.079822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.079784    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a37256d-b608-425e-9b04-0b0db90480b7-kube-api-access-dhnsf" pod "5a37256d-b608-425e-9b04-0b0db90480b7" (UID: "5a37256d-b608-425e-9b04-0b0db90480b7"). InnerVolumeSpecName "kube-api-access-dhnsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:45:53.086678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.080479    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-xks-ca" pod "5a37256d-b608-425e-9b04-0b0db90480b7" (UID: "5a37256d-b608-425e-9b04-0b0db90480b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:45:53.178430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.178340    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a37256d-b608-425e-9b04-0b0db90480b7-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.178430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.178373    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhnsf\" (UniqueName: \"kubernetes.io/projected/5a37256d-b608-425e-9b04-0b0db90480b7-kube-api-access-dhnsf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:45:53.862145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.862109    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"
Oct 30 13:45:53.862145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.862135    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r" event={"ID":"5a37256d-b608-425e-9b04-0b0db90480b7","Type":"ContainerDied","Data":"79feedc74fbffa94ed2508077210065baca7006ca05efca3a20a25fac755dcf0"}
Oct 30 13:45:53.862382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:45:53.862224    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79feedc74fbffa94ed2508077210065baca7006ca05efca3a20a25fac755dcf0"
Oct 30 13:46:00.945843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:00.945806    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"]
Oct 30 13:46:05.035241 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:05.035200    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"]
Oct 30 13:46:05.047090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:05.047053    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-56f1414f-tdzxk"]
Oct 30 13:46:05.907838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:05.907788    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20d96928-a60b-41b8-be36-56fe6a1e8d4e" path="/var/lib/kubelet/pods/20d96928-a60b-41b8-be36-56fe6a1e8d4e/volumes"
Oct 30 13:46:05.946812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:05.946756    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4" podUID="291e6890-c751-4659-b268-239d2c9056e9" containerName="payload-processing" containerID="cri-o://0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f" gracePeriod=25
Oct 30 13:46:06.224510 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.224484    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:46:06.313440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.313401    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-plugins-config-volume\") pod \"291e6890-c751-4659-b268-239d2c9056e9\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") "
Oct 30 13:46:06.313620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.313593    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-kube-api-access-n9pcj\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-kube-api-access-n9pcj\") pod \"291e6890-c751-4659-b268-239d2c9056e9\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") "
Oct 30 13:46:06.313726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.313622    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-routing-overlay-volume\") pod \"291e6890-c751-4659-b268-239d2c9056e9\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") "
Oct 30 13:46:06.313726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.313696    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-provider-credentials\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-provider-credentials\") pod \"291e6890-c751-4659-b268-239d2c9056e9\" (UID: \"291e6890-c751-4659-b268-239d2c9056e9\") "
Oct 30 13:46:06.313833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.313809    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-plugins-config-volume" pod "291e6890-c751-4659-b268-239d2c9056e9" (UID: "291e6890-c751-4659-b268-239d2c9056e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:06.314092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.314063    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-routing-overlay-volume" pod "291e6890-c751-4659-b268-239d2c9056e9" (UID: "291e6890-c751-4659-b268-239d2c9056e9"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:06.314276 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.314258    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.314341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.314282    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/291e6890-c751-4659-b268-239d2c9056e9-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.315911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.315884    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-provider-credentials" pod "291e6890-c751-4659-b268-239d2c9056e9" (UID: "291e6890-c751-4659-b268-239d2c9056e9"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:06.316001 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.315931    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-kube-api-access-n9pcj" pod "291e6890-c751-4659-b268-239d2c9056e9" (UID: "291e6890-c751-4659-b268-239d2c9056e9"). InnerVolumeSpecName "kube-api-access-n9pcj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:06.416224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.416189    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n9pcj\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-kube-api-access-n9pcj\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.416224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.416222    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/291e6890-c751-4659-b268-239d2c9056e9-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:06.808469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.808433    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"]
Oct 30 13:46:06.809127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809110    4291 cpu_manager.go:414] "containerMap: removing container" podUID="291e6890-c751-4659-b268-239d2c9056e9" containerName="payload-processing"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809128    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="291e6890-c751-4659-b268-239d2c9056e9" containerName="payload-processing"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809137    4291 cpu_manager.go:414] "containerMap: removing container" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809142    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809157    4291 cpu_manager.go:414] "containerMap: removing container" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809162    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809173    4291 cpu_manager.go:414] "containerMap: removing container" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809177    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a7ba16-b61c-4139-b1a1-fa2de8b12fa4" containerName="cleanup"
Oct 30 13:46:06.809198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809201    4291 cpu_manager.go:414] "containerMap: removing container" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809207    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809216    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809222    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd06531d-fb03-48a0-864d-b3e33da93784" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809229    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5a37256d-b608-425e-9b04-0b0db90480b7" containerName="revoke-keys"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809234    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a37256d-b608-425e-9b04-0b0db90480b7" containerName="revoke-keys"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809243    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809248    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809257    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809261    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809353    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="291e6890-c751-4659-b268-239d2c9056e9" containerName="payload-processing"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809364    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5aaca5e6-383c-46b3-84b5-b5aa882e37c0" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809373    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="32860ef1-4fcb-45f2-8297-f3330e738442" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809380    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="75451d0a-8ddb-411c-98c7-b4fa2101d048" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809389    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4206617-2601-491e-a274-cb89b62cc103" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809394    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="80678d19-5845-487b-9329-577e2cf9738e" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809400    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="46831069-b252-4c2d-87d4-834789ce4d6e" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809405    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a37256d-b608-425e-9b04-0b0db90480b7" containerName="revoke-keys"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809411    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5714a3ae-1af2-43fd-824c-e6536556a50f" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809416    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7253bff1-99f9-4619-9239-3a76834a05dc" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809423    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="696f6bbf-ec44-4546-8d71-29bd05cf026e" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809430    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="392207f2-fca4-4e13-80b4-218ff561c46f" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809437    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="81d67356-8625-4b93-88fe-b33f21dc9b8e" containerName="cleanup"
Oct 30 13:46:06.809458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.809443    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba738823-8bdf-4597-8eed-7e3a3fb027f2" containerName="cleanup"
Oct 30 13:46:06.813270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.813254    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:06.831741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.831716    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"]
Oct 30 13:46:06.837555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.837523    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"agc-cred-60c0fd6845ba\""
Oct 30 13:46:06.837724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.837576    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"payload-processing-external-model-plugins-e2e-extmod-b-a75d02\""
Oct 30 13:46:06.837724 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.837599    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"kube-root-ca.crt\""
Oct 30 13:46:06.837851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.837736    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"payload-processing-external-model-e2e--08fbcc8d-dockercfg-8k29w\""
Oct 30 13:46:06.840420 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.840398    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"routing-overlay\""
Oct 30 13:46:06.840540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.840424    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-a75d02\"/\"openshift-service-ca.crt\""
Oct 30 13:46:06.963660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.963279    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:06.963660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.963413    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:06.963660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.963493    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2k9k\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-kube-api-access-w2k9k\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:06.964012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:06.963724    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.066716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.066398    4291 generic.go:356] "Generic (PLEG): container finished" podID="291e6890-c751-4659-b268-239d2c9056e9" containerID="0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f" exitCode=0
Oct 30 13:46:07.066886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.066800    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4" event={"ID":"291e6890-c751-4659-b268-239d2c9056e9","Type":"ContainerDied","Data":"0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f"}
Oct 30 13:46:07.066956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.066886    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4" event={"ID":"291e6890-c751-4659-b268-239d2c9056e9","Type":"ContainerDied","Data":"1d027c3af7cae9a73a1a4d0620dc3277b4d42dcc331e5c7e7fbc63435f9b5aa0"}
Oct 30 13:46:07.066956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.066921    4291 scope.go:122] "RemoveContainer" containerID="0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f"
Oct 30 13:46:07.072661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.067164    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"
Oct 30 13:46:07.077547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.077014    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.077547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.077070    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.077547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.077169    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.077547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.077244    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2k9k\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-kube-api-access-w2k9k\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.077547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.077519    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.079683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.078883    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.081368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.081339    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.086979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.086959    4291 scope.go:122] "RemoveContainer" containerID="0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f"
Oct 30 13:46:07.087440 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:46:07.087414    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f\": container with ID starting with 0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f not found: ID does not exist" containerID="0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f"
Oct 30 13:46:07.087596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.087449    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f"} err="failed to get container status \"0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f\": rpc error: code = NotFound desc = could not find container \"0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f\": container with ID starting with 0c6e5d664f0867403eccfca48a7a0a0cd8a03492c42b13c5ee27710d638cc60f not found: ID does not exist"
Oct 30 13:46:07.098279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.098254    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2k9k\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-kube-api-access-w2k9k\") pod \"payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") " pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.128398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.128369    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:07.148386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.148359    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"]
Oct 30 13:46:07.152440 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.152412    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-597f78ch4"]
Oct 30 13:46:07.445306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.445270    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"]
Oct 30 13:46:07.445712 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:46:07.445499    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2b72b0_3038_4cd0_8fa9_2295221fb2fd.slice/crio-009c53b689c9f1306ff31c5ef258b74ed99f18a4c259ed7c75a64163495c8339 WatchSource:0}: Error finding container 009c53b689c9f1306ff31c5ef258b74ed99f18a4c259ed7c75a64163495c8339: Status 404 returned error can't find the container with id 009c53b689c9f1306ff31c5ef258b74ed99f18a4c259ed7c75a64163495c8339
Oct 30 13:46:07.880308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:07.880273    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="291e6890-c751-4659-b268-239d2c9056e9" path="/var/lib/kubelet/pods/291e6890-c751-4659-b268-239d2c9056e9/volumes"
Oct 30 13:46:08.076691 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:08.076655    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm" event={"ID":"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd","Type":"ContainerStarted","Data":"f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d"}
Oct 30 13:46:08.076893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:08.076713    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm" event={"ID":"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd","Type":"ContainerStarted","Data":"009c53b689c9f1306ff31c5ef258b74ed99f18a4c259ed7c75a64163495c8339"}
Oct 30 13:46:08.076893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:08.076758    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:08.132429 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:08.132333    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm" podStartSLOduration=2.132318996 podStartE2EDuration="2.132318996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:08.129277399 +0000 UTC m=+2578.389515634" watchObservedRunningTime="2026-10-30 13:46:08.132318996 +0000 UTC m=+2578.392557230"
Oct 30 13:46:08.211843 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:46:08.211797    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:46:16.045549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:16.045509    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"]
Oct 30 13:46:16.056819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:16.056786    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1eb69cfc-69lwr"]
Oct 30 13:46:17.938658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:17.926498    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0be0a024-6330-453c-b905-8c6094669b07" path="/var/lib/kubelet/pods/0be0a024-6330-453c-b905-8c6094669b07/volumes"
Oct 30 13:46:19.114723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.112845    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:46:19.195843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.195005    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-554987865b-x9dp8"]
Oct 30 13:46:19.204518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.203501    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.221013 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.220980    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-a56ab992d405\""
Oct 30 13:46:19.221725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.221330    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 13:46:19.221895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.221387    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 13:46:19.221980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.221419    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 13:46:19.222060 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.221607    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 13:46:19.222146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.221679    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-nps96\""
Oct 30 13:46:19.225425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.225392    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-554987865b-x9dp8"]
Oct 30 13:46:19.292350 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.292311    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-plugins-config-volume\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.292554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.292533    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-provider-credentials\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.292644 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.292619    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-routing-overlay-volume\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.292876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.292844    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjcc\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-kube-api-access-mqjcc\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.410716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.408981    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-provider-credentials\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.410716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.409098    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-routing-overlay-volume\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.410716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.409169    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjcc\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-kube-api-access-mqjcc\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.410716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.409390    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-plugins-config-volume\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.410716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.410603    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-plugins-config-volume\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.443714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.441412    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-routing-overlay-volume\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.517711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.514207    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjcc\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-kube-api-access-mqjcc\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.557652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.548665    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-provider-credentials\") pod \"payload-processing-external-model-554987865b-x9dp8\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") " pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:19.832571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:19.832484    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:20.154957 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:46:20.154919    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47f884f1_6a51_431e_a46d_0924dff59dac.slice/crio-813cfb6236a213be494faa7097e3226693c35df864b01f24814311ed53a5b9ab WatchSource:0}: Error finding container 813cfb6236a213be494faa7097e3226693c35df864b01f24814311ed53a5b9ab: Status 404 returned error can't find the container with id 813cfb6236a213be494faa7097e3226693c35df864b01f24814311ed53a5b9ab
Oct 30 13:46:20.155774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:20.155744    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-554987865b-x9dp8"]
Oct 30 13:46:20.171677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:20.171628    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8" event={"ID":"47f884f1-6a51-431e-a46d-0924dff59dac","Type":"ContainerStarted","Data":"813cfb6236a213be494faa7097e3226693c35df864b01f24814311ed53a5b9ab"}
Oct 30 13:46:21.181080 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:21.181027    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8" event={"ID":"47f884f1-6a51-431e-a46d-0924dff59dac","Type":"ContainerStarted","Data":"0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa"}
Oct 30 13:46:21.181578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:21.181098    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:21.227074 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:21.227012    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8" podStartSLOduration=2.226997567 podStartE2EDuration="2.226997567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:21.222863019 +0000 UTC m=+2591.483101277" watchObservedRunningTime="2026-10-30 13:46:21.226997567 +0000 UTC m=+2591.487235812"
Oct 30 13:46:24.179940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:24.179902    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-554987865b-x9dp8"]
Oct 30 13:46:28.058261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:28.058220    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"]
Oct 30 13:46:28.062689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:28.062655    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1668c6f-jc5fd"]
Oct 30 13:46:29.191755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.191703    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8" podUID="47f884f1-6a51-431e-a46d-0924dff59dac" containerName="payload-processing" containerID="cri-o://0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa" gracePeriod=25
Oct 30 13:46:29.506423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.506386    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:29.633532 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.633493    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-provider-credentials\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-provider-credentials\") pod \"47f884f1-6a51-431e-a46d-0924dff59dac\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") "
Oct 30 13:46:29.633798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.633601    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-plugins-config-volume\") pod \"47f884f1-6a51-431e-a46d-0924dff59dac\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") "
Oct 30 13:46:29.633798 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.633727    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-routing-overlay-volume\") pod \"47f884f1-6a51-431e-a46d-0924dff59dac\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") "
Oct 30 13:46:29.633937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.633809    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-kube-api-access-mqjcc\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-kube-api-access-mqjcc\") pod \"47f884f1-6a51-431e-a46d-0924dff59dac\" (UID: \"47f884f1-6a51-431e-a46d-0924dff59dac\") "
Oct 30 13:46:29.634076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.634045    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-plugins-config-volume" pod "47f884f1-6a51-431e-a46d-0924dff59dac" (UID: "47f884f1-6a51-431e-a46d-0924dff59dac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:29.634076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.634059    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-routing-overlay-volume" pod "47f884f1-6a51-431e-a46d-0924dff59dac" (UID: "47f884f1-6a51-431e-a46d-0924dff59dac"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:29.634412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.634394    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:29.634496 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.634418    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/47f884f1-6a51-431e-a46d-0924dff59dac-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:29.635937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.635912    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-provider-credentials" pod "47f884f1-6a51-431e-a46d-0924dff59dac" (UID: "47f884f1-6a51-431e-a46d-0924dff59dac"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:29.636053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.635962    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-kube-api-access-mqjcc" pod "47f884f1-6a51-431e-a46d-0924dff59dac" (UID: "47f884f1-6a51-431e-a46d-0924dff59dac"). InnerVolumeSpecName "kube-api-access-mqjcc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:29.768787 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.759867    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:29.768787 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.759904    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjcc\" (UniqueName: \"kubernetes.io/projected/47f884f1-6a51-431e-a46d-0924dff59dac-kube-api-access-mqjcc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:29.910039 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:29.907847    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1" path="/var/lib/kubelet/pods/fdcc5d2d-1d9f-4d8f-8b87-dd43a5953ea1/volumes"
Oct 30 13:46:30.256585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.256552    4291 generic.go:356] "Generic (PLEG): container finished" podID="47f884f1-6a51-431e-a46d-0924dff59dac" containerID="0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa" exitCode=0
Oct 30 13:46:30.257089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.256601    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8"
Oct 30 13:46:30.257089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.256616    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8" event={"ID":"47f884f1-6a51-431e-a46d-0924dff59dac","Type":"ContainerDied","Data":"0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa"}
Oct 30 13:46:30.257089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.256699    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-554987865b-x9dp8" event={"ID":"47f884f1-6a51-431e-a46d-0924dff59dac","Type":"ContainerDied","Data":"813cfb6236a213be494faa7097e3226693c35df864b01f24814311ed53a5b9ab"}
Oct 30 13:46:30.257089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.256723    4291 scope.go:122] "RemoveContainer" containerID="0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa"
Oct 30 13:46:30.267266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.267244    4291 scope.go:122] "RemoveContainer" containerID="0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa"
Oct 30 13:46:30.267609 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:46:30.267587    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa\": container with ID starting with 0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa not found: ID does not exist" containerID="0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa"
Oct 30 13:46:30.267700 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.267619    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa"} err="failed to get container status \"0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa\": rpc error: code = NotFound desc = could not find container \"0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa\": container with ID starting with 0bdb28e44e8018fc060e25e1bc15997685dc586287d3de015a7cc065514191fa not found: ID does not exist"
Oct 30 13:46:30.320612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.320575    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-554987865b-x9dp8"]
Oct 30 13:46:30.343737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:30.343696    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-554987865b-x9dp8"]
Oct 30 13:46:31.898485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:31.895131    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f884f1-6a51-431e-a46d-0924dff59dac" path="/var/lib/kubelet/pods/47f884f1-6a51-431e-a46d-0924dff59dac/volumes"
Oct 30 13:46:34.380297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:34.380255    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:46:38.325244 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:46:38.325184    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:46:39.444678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.444629    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"]
Oct 30 13:46:39.445298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.445283    4291 cpu_manager.go:414] "containerMap: removing container" podUID="47f884f1-6a51-431e-a46d-0924dff59dac" containerName="payload-processing"
Oct 30 13:46:39.445398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.445298    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f884f1-6a51-431e-a46d-0924dff59dac" containerName="payload-processing"
Oct 30 13:46:39.445437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.445430    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f884f1-6a51-431e-a46d-0924dff59dac" containerName="payload-processing"
Oct 30 13:46:39.448997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.448973    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.466619 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.466583    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"]
Oct 30 13:46:39.473249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.473210    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:46:39.473501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.473425    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:46:39.473675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.473572    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:46:39.473760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.473708    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:46:39.526951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.526911    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9g6gc\" (UniqueName: \"kubernetes.io/projected/9f5d8c57-8a0e-49ff-b029-51189c5dc404-kube-api-access-9g6gc\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.526951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.526957    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.527196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.527045    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.628598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.628564    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9g6gc\" (UniqueName: \"kubernetes.io/projected/9f5d8c57-8a0e-49ff-b029-51189c5dc404-kube-api-access-9g6gc\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.628598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.628604    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.628850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.628672    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.629452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.629428    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.631153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.631131    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.650556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.650518    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9g6gc\" (UniqueName: \"kubernetes.io/projected/9f5d8c57-8a0e-49ff-b029-51189c5dc404-kube-api-access-9g6gc\") pod \"maas-api-revoke-sub-bdb85861-gvlqc\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.745746 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.731920    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"]
Oct 30 13:46:39.761705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.761669    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"]
Oct 30 13:46:39.761968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.761780    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.765451 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.765409    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:39.851958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.851915    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9fjm\" (UniqueName: \"kubernetes.io/projected/bca7f724-6a93-4a97-a9cf-994fead4d010-kube-api-access-k9fjm\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.852128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.852048    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.852128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.852106    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.953957 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.953911    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9fjm\" (UniqueName: \"kubernetes.io/projected/bca7f724-6a93-4a97-a9cf-994fead4d010-kube-api-access-k9fjm\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.954180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.954001    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.954180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.954153    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.954778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.954759    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.956607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.956582    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:39.973199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:39.973162    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9fjm\" (UniqueName: \"kubernetes.io/projected/bca7f724-6a93-4a97-a9cf-994fead4d010-kube-api-access-k9fjm\") pod \"maas-api-revoke-sub-d9f6a4f0-t7nb4\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:40.085246 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:40.085213    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"]
Oct 30 13:46:40.085730 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:46:40.085698    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f5d8c57_8a0e_49ff_b029_51189c5dc404.slice/crio-6c308b6eb347e4ac6903d18410243ed4637edfc935c2082b14543f6a17dd9745 WatchSource:0}: Error finding container 6c308b6eb347e4ac6903d18410243ed4637edfc935c2082b14543f6a17dd9745: Status 404 returned error can't find the container with id 6c308b6eb347e4ac6903d18410243ed4637edfc935c2082b14543f6a17dd9745
Oct 30 13:46:40.091215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:40.091194    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:40.375985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:40.375946    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc" event={"ID":"9f5d8c57-8a0e-49ff-b029-51189c5dc404","Type":"ContainerStarted","Data":"6c308b6eb347e4ac6903d18410243ed4637edfc935c2082b14543f6a17dd9745"}
Oct 30 13:46:40.405522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:40.405489    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"]
Oct 30 13:46:40.406426 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:46:40.406394    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca7f724_6a93_4a97_a9cf_994fead4d010.slice/crio-40ec491f0a9f5fb1461485d14cd1f8436efc7e7698fba59d511b141092f03aaf WatchSource:0}: Error finding container 40ec491f0a9f5fb1461485d14cd1f8436efc7e7698fba59d511b141092f03aaf: Status 404 returned error can't find the container with id 40ec491f0a9f5fb1461485d14cd1f8436efc7e7698fba59d511b141092f03aaf
Oct 30 13:46:41.385212 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:41.385171    4291 generic.go:356] "Generic (PLEG): container finished" podID="9f5d8c57-8a0e-49ff-b029-51189c5dc404" containerID="f5459c778debe5d4b0120e14755f9005635dbe969895b15ea3feb40f91b6f7aa" exitCode=0
Oct 30 13:46:41.385678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:41.385256    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc" event={"ID":"9f5d8c57-8a0e-49ff-b029-51189c5dc404","Type":"ContainerDied","Data":"f5459c778debe5d4b0120e14755f9005635dbe969895b15ea3feb40f91b6f7aa"}
Oct 30 13:46:41.386912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:41.386884    4291 generic.go:356] "Generic (PLEG): container finished" podID="bca7f724-6a93-4a97-a9cf-994fead4d010" containerID="93850b2f4e0d79c0b0df545e8d76a43568abbd20e908b96f10843ada46443a99" exitCode=0
Oct 30 13:46:41.387035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:41.386967    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4" event={"ID":"bca7f724-6a93-4a97-a9cf-994fead4d010","Type":"ContainerDied","Data":"93850b2f4e0d79c0b0df545e8d76a43568abbd20e908b96f10843ada46443a99"}
Oct 30 13:46:41.387035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:41.387017    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4" event={"ID":"bca7f724-6a93-4a97-a9cf-994fead4d010","Type":"ContainerStarted","Data":"40ec491f0a9f5fb1461485d14cd1f8436efc7e7698fba59d511b141092f03aaf"}
Oct 30 13:46:42.574549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.574517    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:42.611216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.611184    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:42.691008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.690903    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-openshift-ca\") pod \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") "
Oct 30 13:46:42.691206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.691011    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-xks-ca\") pod \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") "
Oct 30 13:46:42.691206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.691159    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f5d8c57-8a0e-49ff-b029-51189c5dc404-kube-api-access-9g6gc\" (UniqueName: \"kubernetes.io/projected/9f5d8c57-8a0e-49ff-b029-51189c5dc404-kube-api-access-9g6gc\") pod \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\" (UID: \"9f5d8c57-8a0e-49ff-b029-51189c5dc404\") "
Oct 30 13:46:42.691422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.691393    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-openshift-ca" pod "9f5d8c57-8a0e-49ff-b029-51189c5dc404" (UID: "9f5d8c57-8a0e-49ff-b029-51189c5dc404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:42.691719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.691696    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:42.693920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.693890    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-xks-ca" pod "9f5d8c57-8a0e-49ff-b029-51189c5dc404" (UID: "9f5d8c57-8a0e-49ff-b029-51189c5dc404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:42.694041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.693968    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f5d8c57-8a0e-49ff-b029-51189c5dc404-kube-api-access-9g6gc" pod "9f5d8c57-8a0e-49ff-b029-51189c5dc404" (UID: "9f5d8c57-8a0e-49ff-b029-51189c5dc404"). InnerVolumeSpecName "kube-api-access-9g6gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:42.792757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.792719    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-openshift-ca\") pod \"bca7f724-6a93-4a97-a9cf-994fead4d010\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") "
Oct 30 13:46:42.792960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.792841    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca7f724-6a93-4a97-a9cf-994fead4d010-kube-api-access-k9fjm\" (UniqueName: \"kubernetes.io/projected/bca7f724-6a93-4a97-a9cf-994fead4d010-kube-api-access-k9fjm\") pod \"bca7f724-6a93-4a97-a9cf-994fead4d010\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") "
Oct 30 13:46:42.792960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.792899    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-xks-ca\") pod \"bca7f724-6a93-4a97-a9cf-994fead4d010\" (UID: \"bca7f724-6a93-4a97-a9cf-994fead4d010\") "
Oct 30 13:46:42.793169 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.793140    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-openshift-ca" pod "bca7f724-6a93-4a97-a9cf-994fead4d010" (UID: "bca7f724-6a93-4a97-a9cf-994fead4d010"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:42.793391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.793364    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:42.793391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.793386    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f5d8c57-8a0e-49ff-b029-51189c5dc404-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:42.793391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.793396    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9g6gc\" (UniqueName: \"kubernetes.io/projected/9f5d8c57-8a0e-49ff-b029-51189c5dc404-kube-api-access-9g6gc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:42.794955 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.794934    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-xks-ca" pod "bca7f724-6a93-4a97-a9cf-994fead4d010" (UID: "bca7f724-6a93-4a97-a9cf-994fead4d010"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:42.795075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.795050    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca7f724-6a93-4a97-a9cf-994fead4d010-kube-api-access-k9fjm" pod "bca7f724-6a93-4a97-a9cf-994fead4d010" (UID: "bca7f724-6a93-4a97-a9cf-994fead4d010"). InnerVolumeSpecName "kube-api-access-k9fjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:42.895421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.895380    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9fjm\" (UniqueName: \"kubernetes.io/projected/bca7f724-6a93-4a97-a9cf-994fead4d010-kube-api-access-k9fjm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:42.895421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:42.895418    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bca7f724-6a93-4a97-a9cf-994fead4d010-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:43.403489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:43.403457    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"
Oct 30 13:46:43.403489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:43.403473    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc" event={"ID":"9f5d8c57-8a0e-49ff-b029-51189c5dc404","Type":"ContainerDied","Data":"6c308b6eb347e4ac6903d18410243ed4637edfc935c2082b14543f6a17dd9745"}
Oct 30 13:46:43.403756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:43.403519    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c308b6eb347e4ac6903d18410243ed4637edfc935c2082b14543f6a17dd9745"
Oct 30 13:46:43.405046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:43.405024    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"
Oct 30 13:46:43.405194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:43.405061    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4" event={"ID":"bca7f724-6a93-4a97-a9cf-994fead4d010","Type":"ContainerDied","Data":"40ec491f0a9f5fb1461485d14cd1f8436efc7e7698fba59d511b141092f03aaf"}
Oct 30 13:46:43.405194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:43.405104    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40ec491f0a9f5fb1461485d14cd1f8436efc7e7698fba59d511b141092f03aaf"
Oct 30 13:46:47.033667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:47.033616    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"]
Oct 30 13:46:47.045170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:47.045140    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf7a0fe6-77rmq"]
Oct 30 13:46:47.881687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:47.881651    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ae0e243-00fc-4b45-85e9-52d527e5fb25" path="/var/lib/kubelet/pods/4ae0e243-00fc-4b45-85e9-52d527e5fb25/volumes"
Oct 30 13:46:50.305199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.305161    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"]
Oct 30 13:46:50.305829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.305814    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9f5d8c57-8a0e-49ff-b029-51189c5dc404" containerName="revoke-keys"
Oct 30 13:46:50.305829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.305828    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f5d8c57-8a0e-49ff-b029-51189c5dc404" containerName="revoke-keys"
Oct 30 13:46:50.305933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.305864    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bca7f724-6a93-4a97-a9cf-994fead4d010" containerName="revoke-keys"
Oct 30 13:46:50.305933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.305870    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca7f724-6a93-4a97-a9cf-994fead4d010" containerName="revoke-keys"
Oct 30 13:46:50.306019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.305972    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca7f724-6a93-4a97-a9cf-994fead4d010" containerName="revoke-keys"
Oct 30 13:46:50.306019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.305985    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f5d8c57-8a0e-49ff-b029-51189c5dc404" containerName="revoke-keys"
Oct 30 13:46:50.310341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.310321    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.325394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.325364    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"]
Oct 30 13:46:50.331860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.331838    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:46:50.331860 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.331854    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:46:50.332057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.331945    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:46:50.336232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.336207    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:46:50.423852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.423803    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.424045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.424015    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.424095 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.424059    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9s6n\" (UniqueName: \"kubernetes.io/projected/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-kube-api-access-c9s6n\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.525550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.525512    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.525757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.525565    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9s6n\" (UniqueName: \"kubernetes.io/projected/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-kube-api-access-c9s6n\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.525757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.525701    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.526424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.526391    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.528149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.528123    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.544893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.544852    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9s6n\" (UniqueName: \"kubernetes.io/projected/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-kube-api-access-c9s6n\") pod \"maas-api-revoke-sub-4b820a66-jzhg5\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.626389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.626356    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:50.686103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.686050    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"]
Oct 30 13:46:50.692256 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.692227    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.719924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.719887    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"]
Oct 30 13:46:50.730486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.730445    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk2q6\" (UniqueName: \"kubernetes.io/projected/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-kube-api-access-tk2q6\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.731216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.730827    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.731216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.731026    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.832449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.832405    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.832680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.832540    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk2q6\" (UniqueName: \"kubernetes.io/projected/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-kube-api-access-tk2q6\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.832680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.832580    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.833245 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.833216    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.834892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.834870    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.851432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.851404    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk2q6\" (UniqueName: \"kubernetes.io/projected/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-kube-api-access-tk2q6\") pod \"maas-api-revoke-sub-0b37c58c-sqm49\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:50.944411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:50.944381    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"]
Oct 30 13:46:50.946148 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:46:50.946119    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0b3038d_1d24_458c_ad51_6ac9dfe17d85.slice/crio-02544c0e49e405d5208ec71d29650512476ccb531add16c2ce46a128879bae6b WatchSource:0}: Error finding container 02544c0e49e405d5208ec71d29650512476ccb531add16c2ce46a128879bae6b: Status 404 returned error can't find the container with id 02544c0e49e405d5208ec71d29650512476ccb531add16c2ce46a128879bae6b
Oct 30 13:46:51.017513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:51.017475    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:51.325673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:51.325627    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"]
Oct 30 13:46:51.326312 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:46:51.326279    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496f9c7e_6ef7_4b44_b8d5_a7f641af1ffb.slice/crio-4b9869d71a29326e75deaf318b6a79690e26e9e34391a0d791951e2ba8404500 WatchSource:0}: Error finding container 4b9869d71a29326e75deaf318b6a79690e26e9e34391a0d791951e2ba8404500: Status 404 returned error can't find the container with id 4b9869d71a29326e75deaf318b6a79690e26e9e34391a0d791951e2ba8404500
Oct 30 13:46:51.512179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:51.512141    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49" event={"ID":"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb","Type":"ContainerStarted","Data":"4b9869d71a29326e75deaf318b6a79690e26e9e34391a0d791951e2ba8404500"}
Oct 30 13:46:51.513985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:51.513957    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5" event={"ID":"b0b3038d-1d24-458c-ad51-6ac9dfe17d85","Type":"ContainerStarted","Data":"66523595d5d748f0507d8c614bfc50eed4232458943dcc6780a672dfb88f9d5f"}
Oct 30 13:46:51.514139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:51.514012    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5" event={"ID":"b0b3038d-1d24-458c-ad51-6ac9dfe17d85","Type":"ContainerStarted","Data":"02544c0e49e405d5208ec71d29650512476ccb531add16c2ce46a128879bae6b"}
Oct 30 13:46:51.555886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:51.555772    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5" podStartSLOduration=1.5557585729999999 podStartE2EDuration="1.555758573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:46:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:46:51.551382331 +0000 UTC m=+2621.811620576" watchObservedRunningTime="2026-10-30 13:46:51.555758573 +0000 UTC m=+2621.815996806"
Oct 30 13:46:52.523015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:52.522976    4291 generic.go:356] "Generic (PLEG): container finished" podID="b0b3038d-1d24-458c-ad51-6ac9dfe17d85" containerID="66523595d5d748f0507d8c614bfc50eed4232458943dcc6780a672dfb88f9d5f" exitCode=0
Oct 30 13:46:52.523511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:52.523068    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5" event={"ID":"b0b3038d-1d24-458c-ad51-6ac9dfe17d85","Type":"ContainerDied","Data":"66523595d5d748f0507d8c614bfc50eed4232458943dcc6780a672dfb88f9d5f"}
Oct 30 13:46:52.525045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:52.525019    4291 generic.go:356] "Generic (PLEG): container finished" podID="496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" containerID="44f99c5e7a252e1eda50ec32a45e8db42a7316551690f12808b107f881ded1b9" exitCode=0
Oct 30 13:46:52.525168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:52.525070    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49" event={"ID":"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb","Type":"ContainerDied","Data":"44f99c5e7a252e1eda50ec32a45e8db42a7316551690f12808b107f881ded1b9"}
Oct 30 13:46:53.734560 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.734529    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:53.738270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.738244    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:53.780538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.780500    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-kube-api-access-c9s6n\" (UniqueName: \"kubernetes.io/projected/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-kube-api-access-c9s6n\") pod \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") "
Oct 30 13:46:53.780940 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.780907    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-xks-ca\") pod \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") "
Oct 30 13:46:53.781090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.781073    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-kube-api-access-tk2q6\" (UniqueName: \"kubernetes.io/projected/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-kube-api-access-tk2q6\") pod \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") "
Oct 30 13:46:53.781171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.781152    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-xks-ca\") pod \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") "
Oct 30 13:46:53.781403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.781386    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-openshift-ca\") pod \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\" (UID: \"b0b3038d-1d24-458c-ad51-6ac9dfe17d85\") "
Oct 30 13:46:53.781568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.781552    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-openshift-ca\") pod \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\" (UID: \"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb\") "
Oct 30 13:46:53.782596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.782528    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-openshift-ca" pod "496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" (UID: "496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:53.782923 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.782868    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-openshift-ca" pod "b0b3038d-1d24-458c-ad51-6ac9dfe17d85" (UID: "b0b3038d-1d24-458c-ad51-6ac9dfe17d85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:46:53.786732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.786699    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-xks-ca" pod "496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" (UID: "496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:53.787597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.787567    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-kube-api-access-tk2q6" pod "496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" (UID: "496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb"). InnerVolumeSpecName "kube-api-access-tk2q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:53.787748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.787727    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-kube-api-access-c9s6n" pod "b0b3038d-1d24-458c-ad51-6ac9dfe17d85" (UID: "b0b3038d-1d24-458c-ad51-6ac9dfe17d85"). InnerVolumeSpecName "kube-api-access-c9s6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:46:53.789652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.789602    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-xks-ca" pod "b0b3038d-1d24-458c-ad51-6ac9dfe17d85" (UID: "b0b3038d-1d24-458c-ad51-6ac9dfe17d85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:46:53.883088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.883054    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:53.883088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.883083    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk2q6\" (UniqueName: \"kubernetes.io/projected/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-kube-api-access-tk2q6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:53.883088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.883094    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:53.883402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.883104    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:53.883402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.883114    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:53.883402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:53.883122    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9s6n\" (UniqueName: \"kubernetes.io/projected/b0b3038d-1d24-458c-ad51-6ac9dfe17d85-kube-api-access-c9s6n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:46:54.541125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:54.541085    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"
Oct 30 13:46:54.541125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:54.541111    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5" event={"ID":"b0b3038d-1d24-458c-ad51-6ac9dfe17d85","Type":"ContainerDied","Data":"02544c0e49e405d5208ec71d29650512476ccb531add16c2ce46a128879bae6b"}
Oct 30 13:46:54.541405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:54.541186    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02544c0e49e405d5208ec71d29650512476ccb531add16c2ce46a128879bae6b"
Oct 30 13:46:54.542953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:54.542935    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"
Oct 30 13:46:54.543085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:54.542975    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49" event={"ID":"496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb","Type":"ContainerDied","Data":"4b9869d71a29326e75deaf318b6a79690e26e9e34391a0d791951e2ba8404500"}
Oct 30 13:46:54.543085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:46:54.543014    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b9869d71a29326e75deaf318b6a79690e26e9e34391a0d791951e2ba8404500"
Oct 30 13:47:00.034412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:00.034376    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"]
Oct 30 13:47:00.046190 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:00.046158    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cb684c97-r9hmj"]
Oct 30 13:47:01.898163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:01.898124    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9003bb56-77a1-40d8-80ff-5727da7fc74e" path="/var/lib/kubelet/pods/9003bb56-77a1-40d8-80ff-5727da7fc74e/volumes"
Oct 30 13:47:02.093452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.093412    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"]
Oct 30 13:47:02.094085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.094070    4291 cpu_manager.go:414] "containerMap: removing container" podUID="496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" containerName="revoke-keys"
Oct 30 13:47:02.094142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.094087    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" containerName="revoke-keys"
Oct 30 13:47:02.094142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.094107    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b0b3038d-1d24-458c-ad51-6ac9dfe17d85" containerName="revoke-keys"
Oct 30 13:47:02.094142 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.094112    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0b3038d-1d24-458c-ad51-6ac9dfe17d85" containerName="revoke-keys"
Oct 30 13:47:02.094232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.094221    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" containerName="revoke-keys"
Oct 30 13:47:02.094264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.094231    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0b3038d-1d24-458c-ad51-6ac9dfe17d85" containerName="revoke-keys"
Oct 30 13:47:02.099343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.099315    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.117825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.117791    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"]
Oct 30 13:47:02.128682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.125504    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"agc-cred-6feb6713504c\""
Oct 30 13:47:02.128682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.125606    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"routing-overlay\""
Oct 30 13:47:02.128682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.125738    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-e2e--8d6d5ae6-dockercfg-7fjrf\""
Oct 30 13:47:02.128682 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.125818    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"openshift-service-ca.crt\""
Oct 30 13:47:02.129355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.129231    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"payload-processing-external-model-plugins-e2e-extmod-a-cad2c6\""
Oct 30 13:47:02.129355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.129318    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-cad2c6\"/\"kube-root-ca.crt\""
Oct 30 13:47:02.149687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.149572    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.149868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.149825    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.149947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.149894    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.149947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.149937    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4zxj\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-kube-api-access-j4zxj\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.253556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.253500    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.253831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.253574    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4zxj\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-kube-api-access-j4zxj\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.253831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.253778    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.253965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.253914    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.255058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.254786    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.255598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.255563    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.289736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.278347    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.299689 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.298343    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4zxj\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-kube-api-access-j4zxj\") pod \"payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") " pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.421096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.421018    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:02.735462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.735430    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"]
Oct 30 13:47:02.735939 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:47:02.735915    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c85b236_437b_40d0_bf88_acbf25dc025f.slice/crio-e815c8e1111ab7acb8bcaaa94e1debbd1c340d853f4990d5e1e0c75c5969868e WatchSource:0}: Error finding container e815c8e1111ab7acb8bcaaa94e1debbd1c340d853f4990d5e1e0c75c5969868e: Status 404 returned error can't find the container with id e815c8e1111ab7acb8bcaaa94e1debbd1c340d853f4990d5e1e0c75c5969868e
Oct 30 13:47:02.855909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.855873    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"]
Oct 30 13:47:02.863405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.863372    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:02.878260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.878213    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bd2fed\""
Oct 30 13:47:02.879365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.879338    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"]
Oct 30 13:47:02.882613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.882494    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"kube-root-ca.crt\""
Oct 30 13:47:02.882613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.882554    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"agc-cred-acd4b1a0cabb\""
Oct 30 13:47:02.882613 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.882578    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"payload-processing-external-model-e2e--fb32b92e-dockercfg-qkzmw\""
Oct 30 13:47:02.882883 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.882626    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"routing-overlay\""
Oct 30 13:47:02.883066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:02.883047    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"openshift-service-ca.crt\""
Oct 30 13:47:03.082717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.080487    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.082717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.080615    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.082717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.080673    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.082717 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.080952    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5qm9\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-kube-api-access-r5qm9\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.189812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.189742    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5qm9\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-kube-api-access-r5qm9\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.190034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.189879    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.190034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.189961    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.190034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.189994    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.190370 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.190344    4291 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-bd2fed/agc-cred-acd4b1a0cabb: secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:03.190474 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.190370    4291 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt: secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:03.190474 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.190450    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials podName:96370958-75f0-49a9-9ed5-96f517d64a1b nodeName:}" failed. No retries permitted until 2026-10-30 13:47:03.690427391 +0000 UTC m=+2633.950665620 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt" (UID: "96370958-75f0-49a9-9ed5-96f517d64a1b") : secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:03.193754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.193723    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.194434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.194395    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.215395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.215174    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5qm9\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-kube-api-access-r5qm9\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.348236 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.348178    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"]
Oct 30 13:47:03.723768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.720857    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:03.723768 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.721240    4291 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-bd2fed/agc-cred-acd4b1a0cabb: secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:03.723768 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.721260    4291 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt: secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:03.723768 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.721322    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials podName:96370958-75f0-49a9-9ed5-96f517d64a1b nodeName:}" failed. No retries permitted until 2026-10-30 13:47:04.721303132 +0000 UTC m=+2634.981541347 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt" (UID: "96370958-75f0-49a9-9ed5-96f517d64a1b") : secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:03.733658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.729476    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" event={"ID":"5c85b236-437b-40d0-bf88-acbf25dc025f","Type":"ContainerStarted","Data":"b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a"}
Oct 30 13:47:03.733658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.729557    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" event={"ID":"5c85b236-437b-40d0-bf88-acbf25dc025f","Type":"ContainerStarted","Data":"e815c8e1111ab7acb8bcaaa94e1debbd1c340d853f4990d5e1e0c75c5969868e"}
Oct 30 13:47:03.748658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.741882    4291 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" secret="" err="secret \"payload-processing-external-model-e2e--8d6d5ae6-dockercfg-7fjrf\" not found"
Oct 30 13:47:03.748658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:03.742447    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:03.938265 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.936749    4291 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cad2c6/agc-cred-6feb6713504c: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:03.938265 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.936778    4291 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:03.938265 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.936845    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:04.436826762 +0000 UTC m=+2634.697064991 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:03.938265 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.937331    4291 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-plugins-e2e-extmod-a-cad2c6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:03.938265 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:03.937390    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:04.437375108 +0000 UTC m=+2634.697613333 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:04.188889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.188833    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" podStartSLOduration=2.188817517 podStartE2EDuration="2.188817517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:03.80517162 +0000 UTC m=+2634.065409868" watchObservedRunningTime="2026-10-30 13:47:04.188817517 +0000 UTC m=+2634.449055751"
Oct 30 13:47:04.191380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.191350    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"]
Oct 30 13:47:04.191836 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.191806    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt" podUID="96370958-75f0-49a9-9ed5-96f517d64a1b"
Oct 30 13:47:04.443016 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.442935    4291 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-plugins-e2e-extmod-a-cad2c6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:04.443016 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.443009    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:05.442994226 +0000 UTC m=+2635.703232461 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:04.443237 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.443090    4291 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cad2c6/agc-cred-6feb6713504c: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:04.443237 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.443111    4291 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:04.443237 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.443155    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:05.44314261 +0000 UTC m=+2635.703380826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:04.736546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.736457    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:04.745101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.745073    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:04.751966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.751929    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:04.752163 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.752093    4291 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-bd2fed/agc-cred-acd4b1a0cabb: secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:04.752163 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.752115    4291 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt: secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:04.752282 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:04.752201    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials podName:96370958-75f0-49a9-9ed5-96f517d64a1b nodeName:}" failed. No retries permitted until 2026-10-30 13:47:06.752180056 +0000 UTC m=+2637.012418272 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt" (UID: "96370958-75f0-49a9-9ed5-96f517d64a1b") : secret "agc-cred-acd4b1a0cabb" not found
Oct 30 13:47:04.852602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.852566    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-routing-overlay-volume\") pod \"96370958-75f0-49a9-9ed5-96f517d64a1b\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") "
Oct 30 13:47:04.852837 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.852817    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-kube-api-access-r5qm9\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-kube-api-access-r5qm9\") pod \"96370958-75f0-49a9-9ed5-96f517d64a1b\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") "
Oct 30 13:47:04.853010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.852995    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-plugins-config-volume\") pod \"96370958-75f0-49a9-9ed5-96f517d64a1b\" (UID: \"96370958-75f0-49a9-9ed5-96f517d64a1b\") "
Oct 30 13:47:04.853079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.853036    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-routing-overlay-volume" pod "96370958-75f0-49a9-9ed5-96f517d64a1b" (UID: "96370958-75f0-49a9-9ed5-96f517d64a1b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:04.853358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.853337    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-plugins-config-volume" pod "96370958-75f0-49a9-9ed5-96f517d64a1b" (UID: "96370958-75f0-49a9-9ed5-96f517d64a1b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:04.853512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.853494    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:04.853565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.853517    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/96370958-75f0-49a9-9ed5-96f517d64a1b-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:04.855019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.855000    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-kube-api-access-r5qm9" pod "96370958-75f0-49a9-9ed5-96f517d64a1b" (UID: "96370958-75f0-49a9-9ed5-96f517d64a1b"). InnerVolumeSpecName "kube-api-access-r5qm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:04.954471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:04.954434    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5qm9\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-kube-api-access-r5qm9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:05.459420 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:05.459383    4291 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cad2c6/agc-cred-6feb6713504c: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:05.459420 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:05.459413    4291 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:05.459953 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:05.459473    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:07.459458579 +0000 UTC m=+2637.719696805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:05.459953 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:05.459497    4291 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-plugins-e2e-extmod-a-cad2c6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:05.459953 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:05.459553    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:07.459540219 +0000 UTC m=+2637.719778430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:05.743492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:05.743409    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"
Oct 30 13:47:05.823413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:05.823373    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"]
Oct 30 13:47:05.829850 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:05.829796    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-7bdf99jrt"]
Oct 30 13:47:05.864829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:05.864792    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/96370958-75f0-49a9-9ed5-96f517d64a1b-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:05.881732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:05.881695    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96370958-75f0-49a9-9ed5-96f517d64a1b" path="/var/lib/kubelet/pods/96370958-75f0-49a9-9ed5-96f517d64a1b/volumes"
Oct 30 13:47:07.490190 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:07.490147    4291 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-cad2c6/agc-cred-6feb6713504c: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:07.490190 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:07.490184    4291 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4: secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:07.490810 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:07.490620    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:11.490226029 +0000 UTC m=+2641.750464242 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : secret "agc-cred-6feb6713504c" not found
Oct 30 13:47:07.490810 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:07.490700    4291 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-plugins-e2e-extmod-a-cad2c6: configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:07.490810 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:07.490779    4291 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume podName:5c85b236-437b-40d0-bf88-acbf25dc025f nodeName:}" failed. No retries permitted until 2026-10-30 13:47:11.490750129 +0000 UTC m=+2641.750988355 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-cad2c6" not found
Oct 30 13:47:08.595113 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:08.595072    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:47:09.737065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:09.737014    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" podUID="5c85b236-437b-40d0-bf88-acbf25dc025f" containerName="payload-processing" containerID="cri-o://b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a" gracePeriod=25
Oct 30 13:47:10.028835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.028806    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:10.050454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.050418    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-kube-api-access-j4zxj\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-kube-api-access-j4zxj\") pod \"5c85b236-437b-40d0-bf88-acbf25dc025f\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") "
Oct 30 13:47:10.050719 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.050537    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume\") pod \"5c85b236-437b-40d0-bf88-acbf25dc025f\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") "
Oct 30 13:47:10.050802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.050721    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-routing-overlay-volume\") pod \"5c85b236-437b-40d0-bf88-acbf25dc025f\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") "
Oct 30 13:47:10.050802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.050749    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials\") pod \"5c85b236-437b-40d0-bf88-acbf25dc025f\" (UID: \"5c85b236-437b-40d0-bf88-acbf25dc025f\") "
Oct 30 13:47:10.051005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.050969    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume" pod "5c85b236-437b-40d0-bf88-acbf25dc025f" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:10.051275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.051240    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-routing-overlay-volume" pod "5c85b236-437b-40d0-bf88-acbf25dc025f" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:10.051519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.051495    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:10.051610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.051525    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5c85b236-437b-40d0-bf88-acbf25dc025f-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:10.053034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.053004    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-kube-api-access-j4zxj" pod "5c85b236-437b-40d0-bf88-acbf25dc025f" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f"). InnerVolumeSpecName "kube-api-access-j4zxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:10.053034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.053019    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials" pod "5c85b236-437b-40d0-bf88-acbf25dc025f" (UID: "5c85b236-437b-40d0-bf88-acbf25dc025f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:10.153352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.153315    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:10.153352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.153345    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4zxj\" (UniqueName: \"kubernetes.io/projected/5c85b236-437b-40d0-bf88-acbf25dc025f-kube-api-access-j4zxj\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:10.806605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.806564    4291 generic.go:356] "Generic (PLEG): container finished" podID="5c85b236-437b-40d0-bf88-acbf25dc025f" containerID="b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a" exitCode=0
Oct 30 13:47:10.807096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.806614    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"
Oct 30 13:47:10.807096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.806680    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" event={"ID":"5c85b236-437b-40d0-bf88-acbf25dc025f","Type":"ContainerDied","Data":"b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a"}
Oct 30 13:47:10.807096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.806736    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4" event={"ID":"5c85b236-437b-40d0-bf88-acbf25dc025f","Type":"ContainerDied","Data":"e815c8e1111ab7acb8bcaaa94e1debbd1c340d853f4990d5e1e0c75c5969868e"}
Oct 30 13:47:10.807096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.806758    4291 scope.go:122] "RemoveContainer" containerID="b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a"
Oct 30 13:47:10.829585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.818852    4291 scope.go:122] "RemoveContainer" containerID="b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a"
Oct 30 13:47:10.829585 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:10.819322    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a\": container with ID starting with b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a not found: ID does not exist" containerID="b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a"
Oct 30 13:47:10.829585 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.819388    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a"} err="failed to get container status \"b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a\": rpc error: code = NotFound desc = could not find container \"b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a\": container with ID starting with b4dbcdfb7fbe26fe9e4b6e25956fabd913c2e3678a3a3101bb2efc41a47bb60a not found: ID does not exist"
Oct 30 13:47:10.861179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.861144    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"]
Oct 30 13:47:10.873180 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:10.873136    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-cad2c6/payload-processing-external-model-e2e-extmod-a-cad2c6-f8dcj2cb4"]
Oct 30 13:47:11.043773 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:11.043696    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"]
Oct 30 13:47:11.050726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:11.050673    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efa0936c-rslsw"]
Oct 30 13:47:11.880076 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:11.880040    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="042745cf-d3cc-4c6a-a34b-126efbf9b385" path="/var/lib/kubelet/pods/042745cf-d3cc-4c6a-a34b-126efbf9b385/volumes"
Oct 30 13:47:11.880594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:11.880563    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c85b236-437b-40d0-bf88-acbf25dc025f" path="/var/lib/kubelet/pods/5c85b236-437b-40d0-bf88-acbf25dc025f/volumes"
Oct 30 13:47:12.899763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:12.899722    4291 scope.go:122] "RemoveContainer" containerID="b474eae83c75317ff8e25e4aa08f93f4019270208faa6a0bd6daa215e4499030"
Oct 30 13:47:12.910112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:12.910093    4291 scope.go:122] "RemoveContainer" containerID="bdc642dcd79fc9a4a56511dc44e4963e361930a3c08d414e7035295f762a94e7"
Oct 30 13:47:21.963527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.963493    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"]
Oct 30 13:47:21.964219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.964201    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5c85b236-437b-40d0-bf88-acbf25dc025f" containerName="payload-processing"
Oct 30 13:47:21.964262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.964221    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c85b236-437b-40d0-bf88-acbf25dc025f" containerName="payload-processing"
Oct 30 13:47:21.964329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.964319    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c85b236-437b-40d0-bf88-acbf25dc025f" containerName="payload-processing"
Oct 30 13:47:21.969975 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.969947    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:21.991158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.991126    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"]
Oct 30 13:47:21.998172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.998141    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"agc-cred-24361ba6e121\""
Oct 30 13:47:21.998354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.998247    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"kube-root-ca.crt\""
Oct 30 13:47:21.998354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.998264    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"routing-overlay\""
Oct 30 13:47:21.998354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.998331    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"openshift-service-ca.crt\""
Oct 30 13:47:21.998354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.998345    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d2cecb\""
Oct 30 13:47:21.998556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:21.998477    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d2cecb\"/\"payload-processing-external-model-e2e--cb40fe81-dockercfg-z6lx5\""
Oct 30 13:47:22.232720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.228975    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.232720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.229045    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.232720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.229076    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.232720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.229103    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fd6m\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-kube-api-access-2fd6m\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.330557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.330511    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.330793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.330568    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.330793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.330588    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.330793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.330614    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fd6m\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-kube-api-access-2fd6m\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.331338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.331310    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.331454 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.331432    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.333281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.333254    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.350118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.350083    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fd6m\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-kube-api-access-2fd6m\") pod \"payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") " pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.590061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.589962    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:22.903859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.903824    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"]
Oct 30 13:47:22.904479 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:47:22.904447    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33aab158_3aa0_4e83_aaf2_54905aef878b.slice/crio-b03177ac502b75ad3114a6236269210aa4ef49588fdb39af617b60c7cdb3ab1d WatchSource:0}: Error finding container b03177ac502b75ad3114a6236269210aa4ef49588fdb39af617b60c7cdb3ab1d: Status 404 returned error can't find the container with id b03177ac502b75ad3114a6236269210aa4ef49588fdb39af617b60c7cdb3ab1d
Oct 30 13:47:22.940158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:22.940122    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc" event={"ID":"33aab158-3aa0-4e83-aaf2-54905aef878b","Type":"ContainerStarted","Data":"b03177ac502b75ad3114a6236269210aa4ef49588fdb39af617b60c7cdb3ab1d"}
Oct 30 13:47:23.951845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:23.951782    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc" event={"ID":"33aab158-3aa0-4e83-aaf2-54905aef878b","Type":"ContainerStarted","Data":"19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1"}
Oct 30 13:47:23.952493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:23.951912    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:23.993093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:23.993034    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc" podStartSLOduration=2.993019043 podStartE2EDuration="2.993019043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:23.990808239 +0000 UTC m=+2654.251046474" watchObservedRunningTime="2026-10-30 13:47:23.993019043 +0000 UTC m=+2654.253257290"
Oct 30 13:47:26.980054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:26.980014    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:47:34.962858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:34.962826    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:47:38.212082 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:47:38.212042    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:47:49.232707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:49.230737    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-zlnl2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:47:49.232707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:49.230826    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-zlnl2" podUID="da312528-eb75-4d61-89fb-3c2a6ca8c6aa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:47:50.453485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.453447    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"]
Oct 30 13:47:50.458123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.458095    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.476428 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.476393    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"]
Oct 30 13:47:50.483191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.483152    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:47:50.485386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.485360    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:47:50.485628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.485608    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:47:50.485763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.485689    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:47:50.567433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.567388    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4r8zg\" (UniqueName: \"kubernetes.io/projected/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-kube-api-access-4r8zg\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.567666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.567630    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.567770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.567745    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.669277 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.669237    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4r8zg\" (UniqueName: \"kubernetes.io/projected/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-kube-api-access-4r8zg\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.669488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.669373    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.669488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.669407    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.670048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.670022    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.672284 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.672257    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.689771 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.689729    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4r8zg\" (UniqueName: \"kubernetes.io/projected/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-kube-api-access-4r8zg\") pod \"maas-api-revoke-sub-c375bafb-tn72c\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.774436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.774352    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:50.928244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.928205    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"]
Oct 30 13:47:50.932834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.932814    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:50.948566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:50.948532    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"]
Oct 30 13:47:51.075249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.075150    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.075442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.075348    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.075519 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.075497    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8kgt\" (UniqueName: \"kubernetes.io/projected/1d9f2a73-0e1e-4ab7-a692-49a26d640686-kube-api-access-c8kgt\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.079219 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.079195    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"]
Oct 30 13:47:51.080841 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:47:51.080818    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02ae8c03_ffbe_42ff_aab2_e6e71c7a77a1.slice/crio-eec68437ed58278a65fee06c88f6569935c9f414d239ad9d4db363f12516bb0e WatchSource:0}: Error finding container eec68437ed58278a65fee06c88f6569935c9f414d239ad9d4db363f12516bb0e: Status 404 returned error can't find the container with id eec68437ed58278a65fee06c88f6569935c9f414d239ad9d4db363f12516bb0e
Oct 30 13:47:51.176814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.176773    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.177043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.176890    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8kgt\" (UniqueName: \"kubernetes.io/projected/1d9f2a73-0e1e-4ab7-a692-49a26d640686-kube-api-access-c8kgt\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.177043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.176995    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.177741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.177720    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.179309 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.179286    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.196530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.196501    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8kgt\" (UniqueName: \"kubernetes.io/projected/1d9f2a73-0e1e-4ab7-a692-49a26d640686-kube-api-access-c8kgt\") pod \"maas-api-revoke-sub-d8384f11-42wm6\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.250650 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.250613    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:51.341403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.341309    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c" event={"ID":"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1","Type":"ContainerStarted","Data":"eec68437ed58278a65fee06c88f6569935c9f414d239ad9d4db363f12516bb0e"}
Oct 30 13:47:51.562608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:51.562576    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"]
Oct 30 13:47:51.563098 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:47:51.563078    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d9f2a73_0e1e_4ab7_a692_49a26d640686.slice/crio-b081ff4878266af6e674e067dd61112fe2b9ce945ecdb1b495fbff6189008170 WatchSource:0}: Error finding container b081ff4878266af6e674e067dd61112fe2b9ce945ecdb1b495fbff6189008170: Status 404 returned error can't find the container with id b081ff4878266af6e674e067dd61112fe2b9ce945ecdb1b495fbff6189008170
Oct 30 13:47:52.359478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:52.359442    4291 generic.go:356] "Generic (PLEG): container finished" podID="02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" containerID="cfa339140d2fe967f4ddd025af527efa21db08fb730b886ba73e6d7c42c336a9" exitCode=0
Oct 30 13:47:52.359679 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:52.359521    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c" event={"ID":"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1","Type":"ContainerDied","Data":"cfa339140d2fe967f4ddd025af527efa21db08fb730b886ba73e6d7c42c336a9"}
Oct 30 13:47:52.361358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:52.361326    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6" event={"ID":"1d9f2a73-0e1e-4ab7-a692-49a26d640686","Type":"ContainerStarted","Data":"829082cb6aefc3b18a0179519daf08cdd7705345558bb0782e7b60f8dbf427e7"}
Oct 30 13:47:52.361504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:52.361375    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6" event={"ID":"1d9f2a73-0e1e-4ab7-a692-49a26d640686","Type":"ContainerStarted","Data":"b081ff4878266af6e674e067dd61112fe2b9ce945ecdb1b495fbff6189008170"}
Oct 30 13:47:52.439096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:52.439037    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6" podStartSLOduration=2.439021461 podStartE2EDuration="2.439021461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:47:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:47:52.435535635 +0000 UTC m=+2682.695773868" watchObservedRunningTime="2026-10-30 13:47:52.439021461 +0000 UTC m=+2682.699259743"
Oct 30 13:47:53.035511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.035425    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"]
Oct 30 13:47:53.039734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.039706    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ed1e354-bj2xg"]
Oct 30 13:47:53.370125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.370088    4291 generic.go:356] "Generic (PLEG): container finished" podID="1d9f2a73-0e1e-4ab7-a692-49a26d640686" containerID="829082cb6aefc3b18a0179519daf08cdd7705345558bb0782e7b60f8dbf427e7" exitCode=0
Oct 30 13:47:53.370335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.370173    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6" event={"ID":"1d9f2a73-0e1e-4ab7-a692-49a26d640686","Type":"ContainerDied","Data":"829082cb6aefc3b18a0179519daf08cdd7705345558bb0782e7b60f8dbf427e7"}
Oct 30 13:47:53.579132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.579084    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:53.610872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.610838    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-kube-api-access-4r8zg\" (UniqueName: \"kubernetes.io/projected/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-kube-api-access-4r8zg\") pod \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") "
Oct 30 13:47:53.611087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.610890    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-openshift-ca\") pod \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") "
Oct 30 13:47:53.611087 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.611029    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-xks-ca\") pod \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\" (UID: \"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1\") "
Oct 30 13:47:53.611330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.611297    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-openshift-ca" pod "02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" (UID: "02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:53.611763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.611736    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:53.613568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.613537    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-kube-api-access-4r8zg" pod "02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" (UID: "02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1"). InnerVolumeSpecName "kube-api-access-4r8zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:53.613760 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.613541    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-xks-ca" pod "02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" (UID: "02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:47:53.713456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.713412    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4r8zg\" (UniqueName: \"kubernetes.io/projected/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-kube-api-access-4r8zg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:53.713456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.713452    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:53.880164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:53.880074    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a1b052e-5ea8-4fd3-b039-d8e086ef268c" path="/var/lib/kubelet/pods/1a1b052e-5ea8-4fd3-b039-d8e086ef268c/volumes"
Oct 30 13:47:54.379310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.379276    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"
Oct 30 13:47:54.379310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.379298    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c" event={"ID":"02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1","Type":"ContainerDied","Data":"eec68437ed58278a65fee06c88f6569935c9f414d239ad9d4db363f12516bb0e"}
Oct 30 13:47:54.379961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.379352    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eec68437ed58278a65fee06c88f6569935c9f414d239ad9d4db363f12516bb0e"
Oct 30 13:47:54.556413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.556387    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:54.625576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.625535    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d9f2a73-0e1e-4ab7-a692-49a26d640686-kube-api-access-c8kgt\" (UniqueName: \"kubernetes.io/projected/1d9f2a73-0e1e-4ab7-a692-49a26d640686-kube-api-access-c8kgt\") pod \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") "
Oct 30 13:47:54.625805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.625687    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-xks-ca\") pod \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") "
Oct 30 13:47:54.625893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.625831    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-openshift-ca\") pod \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\" (UID: \"1d9f2a73-0e1e-4ab7-a692-49a26d640686\") "
Oct 30 13:47:54.626348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.626313    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-openshift-ca" pod "1d9f2a73-0e1e-4ab7-a692-49a26d640686" (UID: "1d9f2a73-0e1e-4ab7-a692-49a26d640686"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:47:54.626604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.626580    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:54.628104 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.628072    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-xks-ca" pod "1d9f2a73-0e1e-4ab7-a692-49a26d640686" (UID: "1d9f2a73-0e1e-4ab7-a692-49a26d640686"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:47:54.628790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.628761    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d9f2a73-0e1e-4ab7-a692-49a26d640686-kube-api-access-c8kgt" pod "1d9f2a73-0e1e-4ab7-a692-49a26d640686" (UID: "1d9f2a73-0e1e-4ab7-a692-49a26d640686"). InnerVolumeSpecName "kube-api-access-c8kgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:47:54.728202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.728148    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8kgt\" (UniqueName: \"kubernetes.io/projected/1d9f2a73-0e1e-4ab7-a692-49a26d640686-kube-api-access-c8kgt\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:54.728202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:54.728200    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d9f2a73-0e1e-4ab7-a692-49a26d640686-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:47:55.387904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:55.387871    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"
Oct 30 13:47:55.388336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:55.387881    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6" event={"ID":"1d9f2a73-0e1e-4ab7-a692-49a26d640686","Type":"ContainerDied","Data":"b081ff4878266af6e674e067dd61112fe2b9ce945ecdb1b495fbff6189008170"}
Oct 30 13:47:55.388336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:47:55.387995    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b081ff4878266af6e674e067dd61112fe2b9ce945ecdb1b495fbff6189008170"
Oct 30 13:48:02.802285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:02.802230    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"]
Oct 30 13:48:03.040024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.039988    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"]
Oct 30 13:48:03.040852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.040830    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1d9f2a73-0e1e-4ab7-a692-49a26d640686" containerName="revoke-keys"
Oct 30 13:48:03.040852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.040851    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d9f2a73-0e1e-4ab7-a692-49a26d640686" containerName="revoke-keys"
Oct 30 13:48:03.041053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.040882    4291 cpu_manager.go:414] "containerMap: removing container" podUID="02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" containerName="revoke-keys"
Oct 30 13:48:03.041053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.040890    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" containerName="revoke-keys"
Oct 30 13:48:03.041157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.041081    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d9f2a73-0e1e-4ab7-a692-49a26d640686" containerName="revoke-keys"
Oct 30 13:48:03.041157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.041102    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" containerName="revoke-keys"
Oct 30 13:48:03.045038 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.045015    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.065840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.065619    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:48:03.065840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.065743    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:48:03.065840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.065781    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:48:03.065840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.065809    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:48:03.066216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.065874    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"]
Oct 30 13:48:03.204659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.187597    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.204659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.187829    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.231318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.230891    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnqjg\" (UniqueName: \"kubernetes.io/projected/90ec351a-c373-4b50-9632-1d7affadb8b1-kube-api-access-gnqjg\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.332971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.332419    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.332971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.332654    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.332971 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.332770    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnqjg\" (UniqueName: \"kubernetes.io/projected/90ec351a-c373-4b50-9632-1d7affadb8b1-kube-api-access-gnqjg\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.334035 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.334008    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.339433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.339389    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.352240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.352212    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnqjg\" (UniqueName: \"kubernetes.io/projected/90ec351a-c373-4b50-9632-1d7affadb8b1-kube-api-access-gnqjg\") pod \"maas-api-revoke-sub-83166978-m2nj2\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.360058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.360030    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:03.685386 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:48:03.685349    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ec351a_c373_4b50_9632_1d7affadb8b1.slice/crio-616f5125a543e83d5f38ed0f3cea4c752f3a1ce9ef24dfdff1606d1dae8e7582 WatchSource:0}: Error finding container 616f5125a543e83d5f38ed0f3cea4c752f3a1ce9ef24dfdff1606d1dae8e7582: Status 404 returned error can't find the container with id 616f5125a543e83d5f38ed0f3cea4c752f3a1ce9ef24dfdff1606d1dae8e7582
Oct 30 13:48:03.686571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:03.686548    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"]
Oct 30 13:48:04.589936 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:04.589905    4291 generic.go:356] "Generic (PLEG): container finished" podID="90ec351a-c373-4b50-9632-1d7affadb8b1" containerID="264c8b57daf89855959c374736f2d47bf6fa079f1e11894e194cde2a6b8fb26b" exitCode=0
Oct 30 13:48:04.590388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:04.589999    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2" event={"ID":"90ec351a-c373-4b50-9632-1d7affadb8b1","Type":"ContainerDied","Data":"264c8b57daf89855959c374736f2d47bf6fa079f1e11894e194cde2a6b8fb26b"}
Oct 30 13:48:04.590388 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:04.590057    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2" event={"ID":"90ec351a-c373-4b50-9632-1d7affadb8b1","Type":"ContainerStarted","Data":"616f5125a543e83d5f38ed0f3cea4c752f3a1ce9ef24dfdff1606d1dae8e7582"}
Oct 30 13:48:05.753453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.753429    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:05.877088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.876997    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-openshift-ca\") pod \"90ec351a-c373-4b50-9632-1d7affadb8b1\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") "
Oct 30 13:48:05.877088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.877045    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90ec351a-c373-4b50-9632-1d7affadb8b1-kube-api-access-gnqjg\" (UniqueName: \"kubernetes.io/projected/90ec351a-c373-4b50-9632-1d7affadb8b1-kube-api-access-gnqjg\") pod \"90ec351a-c373-4b50-9632-1d7affadb8b1\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") "
Oct 30 13:48:05.877355 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.877332    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-xks-ca\") pod \"90ec351a-c373-4b50-9632-1d7affadb8b1\" (UID: \"90ec351a-c373-4b50-9632-1d7affadb8b1\") "
Oct 30 13:48:05.877455 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.877432    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-openshift-ca" pod "90ec351a-c373-4b50-9632-1d7affadb8b1" (UID: "90ec351a-c373-4b50-9632-1d7affadb8b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:05.878125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.878097    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:05.879947 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.879920    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90ec351a-c373-4b50-9632-1d7affadb8b1-kube-api-access-gnqjg" pod "90ec351a-c373-4b50-9632-1d7affadb8b1" (UID: "90ec351a-c373-4b50-9632-1d7affadb8b1"). InnerVolumeSpecName "kube-api-access-gnqjg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:05.882156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.882130    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-xks-ca" pod "90ec351a-c373-4b50-9632-1d7affadb8b1" (UID: "90ec351a-c373-4b50-9632-1d7affadb8b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:48:05.979342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.979304    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnqjg\" (UniqueName: \"kubernetes.io/projected/90ec351a-c373-4b50-9632-1d7affadb8b1-kube-api-access-gnqjg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:05.979342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:05.979340    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90ec351a-c373-4b50-9632-1d7affadb8b1-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:06.649666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:06.647416    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2" event={"ID":"90ec351a-c373-4b50-9632-1d7affadb8b1","Type":"ContainerDied","Data":"616f5125a543e83d5f38ed0f3cea4c752f3a1ce9ef24dfdff1606d1dae8e7582"}
Oct 30 13:48:06.649666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:06.647480    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="616f5125a543e83d5f38ed0f3cea4c752f3a1ce9ef24dfdff1606d1dae8e7582"
Oct 30 13:48:06.649666 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:06.647550    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"
Oct 30 13:48:07.803554 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:07.803479    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm" podUID="ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" containerName="payload-processing" containerID="cri-o://f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d" gracePeriod=25
Oct 30 13:48:08.222662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.211036    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:48:08.263745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.261596    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-routing-overlay-volume\") pod \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") "
Oct 30 13:48:08.263745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.261705    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-kube-api-access-w2k9k\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-kube-api-access-w2k9k\") pod \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") "
Oct 30 13:48:08.263745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.261891    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-provider-credentials\") pod \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") "
Oct 30 13:48:08.263745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.261937    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-plugins-config-volume\") pod \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\" (UID: \"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd\") "
Oct 30 13:48:08.263745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.263428    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-routing-overlay-volume" pod "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" (UID: "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:08.264436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.264393    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-plugins-config-volume" pod "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" (UID: "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:08.306488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.306218    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-provider-credentials" pod "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" (UID: "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:08.306868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.306825    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-kube-api-access-w2k9k" pod "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" (UID: "ff2b72b0-3038-4cd0-8fa9-2295221fb2fd"). InnerVolumeSpecName "kube-api-access-w2k9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:08.347378 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:48:08.344589    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:48:08.364336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.364240    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:08.364336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.364268    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:08.364336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.364280    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:08.364336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.364297    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2k9k\" (UniqueName: \"kubernetes.io/projected/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd-kube-api-access-w2k9k\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:08.663381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.663287    4291 generic.go:356] "Generic (PLEG): container finished" podID="ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" containerID="f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d" exitCode=0
Oct 30 13:48:08.663381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.663333    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"
Oct 30 13:48:08.663381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.663368    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm" event={"ID":"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd","Type":"ContainerDied","Data":"f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d"}
Oct 30 13:48:08.663618 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.663426    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm" event={"ID":"ff2b72b0-3038-4cd0-8fa9-2295221fb2fd","Type":"ContainerDied","Data":"009c53b689c9f1306ff31c5ef258b74ed99f18a4c259ed7c75a64163495c8339"}
Oct 30 13:48:08.663618 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.663449    4291 scope.go:122] "RemoveContainer" containerID="f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d"
Oct 30 13:48:08.675505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.675483    4291 scope.go:122] "RemoveContainer" containerID="f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d"
Oct 30 13:48:08.675844 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:48:08.675822    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d\": container with ID starting with f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d not found: ID does not exist" containerID="f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d"
Oct 30 13:48:08.675935 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.675853    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d"} err="failed to get container status \"f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d\": rpc error: code = NotFound desc = could not find container \"f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d\": container with ID starting with f71f08a3b03c1714b3b25f99b4456d4cbce11a6859ff344d62bdc1558f81f98d not found: ID does not exist"
Oct 30 13:48:08.727683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.727624    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"]
Oct 30 13:48:08.742157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:08.742121    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-a75d02/payload-processing-external-model-e2e-extmod-b-a75d02-b9bcrtqsm"]
Oct 30 13:48:09.881669 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:09.881613    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" path="/var/lib/kubelet/pods/ff2b72b0-3038-4cd0-8fa9-2295221fb2fd/volumes"
Oct 30 13:48:12.958163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:12.958128    4291 scope.go:122] "RemoveContainer" containerID="5c48243fcfcc1789f370f48bf15f6f9c4471c9104953c7dd16a0a4dcc28a5127"
Oct 30 13:48:12.968838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:12.968805    4291 scope.go:122] "RemoveContainer" containerID="663d1c1bc427c3f7af3d59f000b9ecaadfda862fe9d434fb2ab5eaf4e99718e7"
Oct 30 13:48:12.979383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:12.979358    4291 scope.go:122] "RemoveContainer" containerID="624bc313313d9a36d726e8d829d1045c9f76bba1d23d7579d98763904b7e73cf"
Oct 30 13:48:12.989416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:12.989396    4291 scope.go:122] "RemoveContainer" containerID="60ecec4c121157fd9176bc157d020d4d41da2617a928ea62774fd2f67b07373b"
Oct 30 13:48:14.924314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:14.924279    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"]
Oct 30 13:48:19.925583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:19.925525    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws" podUID="7dc309c7-1d98-49f1-a0bd-e22b351c1178" containerName="payload-processing" containerID="cri-o://2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d" gracePeriod=25
Oct 30 13:48:20.200141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.200109    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:48:20.292291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.292250    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-plugins-config-volume\") pod \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") "
Oct 30 13:48:20.292291 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.292295    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-kube-api-access-5d5wn\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-kube-api-access-5d5wn\") pod \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") "
Oct 30 13:48:20.292598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.292344    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-provider-credentials\") pod \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") "
Oct 30 13:48:20.292598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.292482    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-routing-overlay-volume\") pod \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\" (UID: \"7dc309c7-1d98-49f1-a0bd-e22b351c1178\") "
Oct 30 13:48:20.292748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.292726    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-plugins-config-volume" pod "7dc309c7-1d98-49f1-a0bd-e22b351c1178" (UID: "7dc309c7-1d98-49f1-a0bd-e22b351c1178"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:20.292910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.292888    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-routing-overlay-volume" pod "7dc309c7-1d98-49f1-a0bd-e22b351c1178" (UID: "7dc309c7-1d98-49f1-a0bd-e22b351c1178"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:20.293393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.293372    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:20.293443 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.293403    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/7dc309c7-1d98-49f1-a0bd-e22b351c1178-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:20.294831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.294805    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-provider-credentials" pod "7dc309c7-1d98-49f1-a0bd-e22b351c1178" (UID: "7dc309c7-1d98-49f1-a0bd-e22b351c1178"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:20.294945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.294828    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-kube-api-access-5d5wn" pod "7dc309c7-1d98-49f1-a0bd-e22b351c1178" (UID: "7dc309c7-1d98-49f1-a0bd-e22b351c1178"). InnerVolumeSpecName "kube-api-access-5d5wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:20.394779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.394734    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5d5wn\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-kube-api-access-5d5wn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:20.394779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.394770    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/7dc309c7-1d98-49f1-a0bd-e22b351c1178-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:20.814027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.813987    4291 generic.go:356] "Generic (PLEG): container finished" podID="7dc309c7-1d98-49f1-a0bd-e22b351c1178" containerID="2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d" exitCode=0
Oct 30 13:48:20.814234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.814037    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"
Oct 30 13:48:20.814234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.814070    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws" event={"ID":"7dc309c7-1d98-49f1-a0bd-e22b351c1178","Type":"ContainerDied","Data":"2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d"}
Oct 30 13:48:20.814234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.814131    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws" event={"ID":"7dc309c7-1d98-49f1-a0bd-e22b351c1178","Type":"ContainerDied","Data":"8693ba91737b83c0073c2b77f8cf6089e7f15d96efc84771fdc48922b19671a0"}
Oct 30 13:48:20.814234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.814155    4291 scope.go:122] "RemoveContainer" containerID="2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d"
Oct 30 13:48:20.825653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.825620    4291 scope.go:122] "RemoveContainer" containerID="2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d"
Oct 30 13:48:20.825969 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:48:20.825945    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d\": container with ID starting with 2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d not found: ID does not exist" containerID="2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d"
Oct 30 13:48:20.826050 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.825979    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d"} err="failed to get container status \"2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d\": rpc error: code = NotFound desc = could not find container \"2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d\": container with ID starting with 2dd6d82fc9a7267935b53beba5588d30befc2336a2d6cb81ef016429d5c4e84d not found: ID does not exist"
Oct 30 13:48:20.867278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.867220    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"]
Oct 30 13:48:20.878530 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:20.878481    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-5b76wwsws"]
Oct 30 13:48:21.652416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:21.652378    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:48:21.880304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:21.880269    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc309c7-1d98-49f1-a0bd-e22b351c1178" path="/var/lib/kubelet/pods/7dc309c7-1d98-49f1-a0bd-e22b351c1178/volumes"
Oct 30 13:48:26.042758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.042712    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"]
Oct 30 13:48:26.054358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.054318    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6493b38-97zp4"]
Oct 30 13:48:26.332557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.332457    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"]
Oct 30 13:48:26.333194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333172    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" containerName="payload-processing"
Oct 30 13:48:26.333319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333196    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" containerName="payload-processing"
Oct 30 13:48:26.333319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333222    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7dc309c7-1d98-49f1-a0bd-e22b351c1178" containerName="payload-processing"
Oct 30 13:48:26.333319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333231    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc309c7-1d98-49f1-a0bd-e22b351c1178" containerName="payload-processing"
Oct 30 13:48:26.333319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333291    4291 cpu_manager.go:414] "containerMap: removing container" podUID="90ec351a-c373-4b50-9632-1d7affadb8b1" containerName="revoke-keys"
Oct 30 13:48:26.333319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333300    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90ec351a-c373-4b50-9632-1d7affadb8b1" containerName="revoke-keys"
Oct 30 13:48:26.333573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333472    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="90ec351a-c373-4b50-9632-1d7affadb8b1" containerName="revoke-keys"
Oct 30 13:48:26.333573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333489    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2b72b0-3038-4cd0-8fa9-2295221fb2fd" containerName="payload-processing"
Oct 30 13:48:26.333573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.333502    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc309c7-1d98-49f1-a0bd-e22b351c1178" containerName="payload-processing"
Oct 30 13:48:26.338748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.338725    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.362424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.362382    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"]
Oct 30 13:48:26.366039 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.366008    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c3efb2\""
Oct 30 13:48:26.366341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.366314    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"openshift-service-ca.crt\""
Oct 30 13:48:26.366477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.366398    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"kube-root-ca.crt\""
Oct 30 13:48:26.366477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.366424    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"payload-processing-external-model-e2e--2f6d7ae8-dockercfg-7lf2v\""
Oct 30 13:48:26.366595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.366483    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"agc-cred-7218a16ac783\""
Oct 30 13:48:26.366595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.366497    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c3efb2\"/\"routing-overlay\""
Oct 30 13:48:26.370000 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.369963    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.370148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.370034    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbv9f\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-kube-api-access-zbv9f\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.370148 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.370115    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.370280 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.370212    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.488575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.488523    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.490140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.490111    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.491138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.491113    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.492030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.492010    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbv9f\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-kube-api-access-zbv9f\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.492624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.491931    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.492851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.489966    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.520336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.520298    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.545710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.545676    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbv9f\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-kube-api-access-zbv9f\") pod \"payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") " pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:26.655093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:26.655056    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:27.025011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:27.024972    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"]
Oct 30 13:48:27.026221 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:48:27.026191    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1d5b1ea_93cc_48a2_aa8c_fdb923137d63.slice/crio-42740016ab4adc2c336b7040e06c9c41bfd54037f5f42832cf85d6ad07bbed11 WatchSource:0}: Error finding container 42740016ab4adc2c336b7040e06c9c41bfd54037f5f42832cf85d6ad07bbed11: Status 404 returned error can't find the container with id 42740016ab4adc2c336b7040e06c9c41bfd54037f5f42832cf85d6ad07bbed11
Oct 30 13:48:28.067672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:28.066040    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98ae2010-d727-4e48-a71b-0082a5005a62" path="/var/lib/kubelet/pods/98ae2010-d727-4e48-a71b-0082a5005a62/volumes"
Oct 30 13:48:28.080150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:28.078822    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb" event={"ID":"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63","Type":"ContainerStarted","Data":"ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e"}
Oct 30 13:48:28.080150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:28.078894    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb" event={"ID":"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63","Type":"ContainerStarted","Data":"42740016ab4adc2c336b7040e06c9c41bfd54037f5f42832cf85d6ad07bbed11"}
Oct 30 13:48:28.080150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:28.080081    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:29.729098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.729041    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb" podStartSLOduration=3.729025581 podStartE2EDuration="3.729025581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:28.126102169 +0000 UTC m=+2718.386340407" watchObservedRunningTime="2026-10-30 13:48:29.729025581 +0000 UTC m=+2719.989263815"
Oct 30 13:48:29.730919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.730898    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"]
Oct 30 13:48:29.735115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.735097    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.758562 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.758517    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"]
Oct 30 13:48:29.761307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.760767    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"agc-cred-739de64450c0\""
Oct 30 13:48:29.761307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.760817    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"agc-cred-e539133cb1f7\""
Oct 30 13:48:29.761307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.761057    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"openshift-service-ca.crt\""
Oct 30 13:48:29.761307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.761086    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"kube-root-ca.crt\""
Oct 30 13:48:29.761307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.761154    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"payload-processing-external-model-e2e--8b865544-dockercfg-rbmwm\""
Oct 30 13:48:29.768750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.768721    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"payload-processing-external-model-plugins-e2e-extmod-b-98f1cc\""
Oct 30 13:48:29.768911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.768791    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-98f1cc\"/\"routing-overlay\""
Oct 30 13:48:29.843473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.843436    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.843703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.843563    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.843703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.843672    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xltpn\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-kube-api-access-xltpn\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.843792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.843778    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.945395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.945344    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.945743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.945458    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.945743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.945518    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xltpn\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-kube-api-access-xltpn\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.945743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.945581    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.946500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.946474    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.949685 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.946922    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.964569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.964532    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:29.966305 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:29.966275    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xltpn\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-kube-api-access-xltpn\") pod \"payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") " pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:30.051896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:30.051802    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:30.362358 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:30.362326    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"]
Oct 30 13:48:30.363622 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:48:30.363585    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6f5c474_4e88_4eb2_ba77_2d9e7658c150.slice/crio-1f4fe68e2bc137e6da7e5fb1712d6362a564c28114c4fb291018ba6b221039e2 WatchSource:0}: Error finding container 1f4fe68e2bc137e6da7e5fb1712d6362a564c28114c4fb291018ba6b221039e2: Status 404 returned error can't find the container with id 1f4fe68e2bc137e6da7e5fb1712d6362a564c28114c4fb291018ba6b221039e2
Oct 30 13:48:31.109604 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.109567    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9" event={"ID":"d6f5c474-4e88-4eb2-ba77-2d9e7658c150","Type":"ContainerStarted","Data":"12fd99b10d266618d954cfa0cee887954d567b0d79b92fcd4285de7b38f4e4ef"}
Oct 30 13:48:31.110160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.109631    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9" event={"ID":"d6f5c474-4e88-4eb2-ba77-2d9e7658c150","Type":"ContainerStarted","Data":"1f4fe68e2bc137e6da7e5fb1712d6362a564c28114c4fb291018ba6b221039e2"}
Oct 30 13:48:31.110160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.109815    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:31.152441 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.152330    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9" podStartSLOduration=2.152308797 podStartE2EDuration="2.152308797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:31.147803684 +0000 UTC m=+2721.408041932" watchObservedRunningTime="2026-10-30 13:48:31.152308797 +0000 UTC m=+2721.412547032"
Oct 30 13:48:31.330216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.330173    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"]
Oct 30 13:48:31.336164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.336131    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.355359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.354794    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-5494c9-dockercfg-gkxfq\""
Oct 30 13:48:31.355853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.355823    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"]
Oct 30 13:48:31.368086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.368000    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrff7\" (UniqueName: \"kubernetes.io/projected/c82b7230-22c5-476d-8452-1c7f301e5630-kube-api-access-nrff7\") pod \"payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.368086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.368053    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b7230-22c5-476d-8452-1c7f301e5630-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.449828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.433945    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-5494c9\""
Oct 30 13:48:31.482659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.474579    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrff7\" (UniqueName: \"kubernetes.io/projected/c82b7230-22c5-476d-8452-1c7f301e5630-kube-api-access-nrff7\") pod \"payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.482659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.474690    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b7230-22c5-476d-8452-1c7f301e5630-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.482659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.476411    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b7230-22c5-476d-8452-1c7f301e5630-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.523503 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.523464    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrff7\" (UniqueName: \"kubernetes.io/projected/c82b7230-22c5-476d-8452-1c7f301e5630-kube-api-access-nrff7\") pod \"payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") " pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.674712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.673672    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:31.714392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:31.713251    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"]
Oct 30 13:48:32.171149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:32.168501    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"]
Oct 30 13:48:33.194805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:33.194770    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" event={"ID":"c82b7230-22c5-476d-8452-1c7f301e5630","Type":"ContainerStarted","Data":"b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0"}
Oct 30 13:48:33.195257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:33.194826    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" event={"ID":"c82b7230-22c5-476d-8452-1c7f301e5630","Type":"ContainerStarted","Data":"54cb177aca509a661f246fbdd0af88ce9469a66d6d7ae73fbabac28cd5b00260"}
Oct 30 13:48:33.195257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:33.194864    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:33.236506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:33.236453    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" podStartSLOduration=2.236439817 podStartE2EDuration="2.236439817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:33.233802559 +0000 UTC m=+2723.494040794" watchObservedRunningTime="2026-10-30 13:48:33.236439817 +0000 UTC m=+2723.496678050"
Oct 30 13:48:36.714081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:36.714027    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb" podUID="d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" containerName="payload-processing" containerID="cri-o://ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e" gracePeriod=25
Oct 30 13:48:37.010304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.010278    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:37.139066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139027    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-kube-api-access-zbv9f\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-kube-api-access-zbv9f\") pod \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") "
Oct 30 13:48:37.139286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139096    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-provider-credentials\") pod \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") "
Oct 30 13:48:37.139286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139131    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-plugins-config-volume\") pod \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") "
Oct 30 13:48:37.139431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139364    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-routing-overlay-volume\") pod \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\" (UID: \"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63\") "
Oct 30 13:48:37.139549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139523    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-plugins-config-volume" pod "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" (UID: "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:37.139764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-routing-overlay-volume" pod "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" (UID: "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:48:37.139930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139913    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:37.139980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.139933    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:37.141908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.141884    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-provider-credentials" pod "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" (UID: "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:37.142020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.141919    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-kube-api-access-zbv9f" pod "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" (UID: "d1d5b1ea-93cc-48a2-aa8c-fdb923137d63"). InnerVolumeSpecName "kube-api-access-zbv9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:48:37.226792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.226760    4291 generic.go:356] "Generic (PLEG): container finished" podID="d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" containerID="ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e" exitCode=0
Oct 30 13:48:37.226978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.226805    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"
Oct 30 13:48:37.226978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.226853    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb" event={"ID":"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63","Type":"ContainerDied","Data":"ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e"}
Oct 30 13:48:37.226978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.226907    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb" event={"ID":"d1d5b1ea-93cc-48a2-aa8c-fdb923137d63","Type":"ContainerDied","Data":"42740016ab4adc2c336b7040e06c9c41bfd54037f5f42832cf85d6ad07bbed11"}
Oct 30 13:48:37.226978 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.226928    4291 scope.go:122] "RemoveContainer" containerID="ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e"
Oct 30 13:48:37.242822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.242787    4291 scope.go:122] "RemoveContainer" containerID="ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e"
Oct 30 13:48:37.244060 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:48:37.244028    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e\": container with ID starting with ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e not found: ID does not exist" containerID="ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e"
Oct 30 13:48:37.244224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.244068    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e"} err="failed to get container status \"ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e\": rpc error: code = NotFound desc = could not find container \"ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e\": container with ID starting with ea6efec8b0851ced66a835b1192b7b249738ff6c21be14df01c5e603bf98965e not found: ID does not exist"
Oct 30 13:48:37.244224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.244137    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbv9f\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-kube-api-access-zbv9f\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:37.244224 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.244162    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:48:37.325116 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.325079    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"]
Oct 30 13:48:37.329281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.329250    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c3efb2/payload-processing-external-model-e2e-extmod-a-c3efb2-79462l8xb"]
Oct 30 13:48:37.881745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:37.881705    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" path="/var/lib/kubelet/pods/d1d5b1ea-93cc-48a2-aa8c-fdb923137d63/volumes"
Oct 30 13:48:38.038417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:38.038379    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"]
Oct 30 13:48:38.049790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:38.049754    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd7ac126-kzgrw"]
Oct 30 13:48:38.213911 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:48:38.213821    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:48:39.881528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:39.881491    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f94f1fec-4ad4-4699-84a4-2bb25df26896" path="/var/lib/kubelet/pods/f94f1fec-4ad4-4699-84a4-2bb25df26896/volumes"
Oct 30 13:48:42.206881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:42.206852    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:48:44.241599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:44.241558    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:48:47.166311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.166272    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"]
Oct 30 13:48:47.166949 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.166933    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" containerName="payload-processing"
Oct 30 13:48:47.167007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.166950    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" containerName="payload-processing"
Oct 30 13:48:47.167096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.167086    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1d5b1ea-93cc-48a2-aa8c-fdb923137d63" containerName="payload-processing"
Oct 30 13:48:47.172088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.172063    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.197725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.197691    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"]
Oct 30 13:48:47.202779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.202743    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-51605e-dockercfg-5tkc4\""
Oct 30 13:48:47.202779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.202743    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-51605e\""
Oct 30 13:48:47.208533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.208496    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxz9n\" (UniqueName: \"kubernetes.io/projected/47f9a857-ba8b-4c06-bd76-faa29e505dbf-kube-api-access-mxz9n\") pod \"payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.208802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.208773    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f9a857-ba8b-4c06-bd76-faa29e505dbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.391610 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.391572    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxz9n\" (UniqueName: \"kubernetes.io/projected/47f9a857-ba8b-4c06-bd76-faa29e505dbf-kube-api-access-mxz9n\") pod \"payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.393614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.393579    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f9a857-ba8b-4c06-bd76-faa29e505dbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.394854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.394709    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f9a857-ba8b-4c06-bd76-faa29e505dbf-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.424550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.424470    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxz9n\" (UniqueName: \"kubernetes.io/projected/47f9a857-ba8b-4c06-bd76-faa29e505dbf-kube-api-access-mxz9n\") pod \"payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.504728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.504689    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:47.520360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.520322    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"]
Oct 30 13:48:47.526761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.526734    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.546341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.546301    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"]
Oct 30 13:48:47.587216 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.587174    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-844856-openshift-default-dockercfg-wmrs2\""
Oct 30 13:48:47.600493 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600459    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znrr5\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-kube-api-access-znrr5\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.600732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600512    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-token\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.600732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600545    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-envoy\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.600732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600589    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-certs\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.600732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600626    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-credential-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.600973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600796    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-podinfo\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.600973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600910    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e9f923-8951-4f90-becb-fc85ff30a41b-istiod-ca-cert\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.601081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.600991    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.601081 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.601035    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-data\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.702946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.702828    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-podinfo\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.702982    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e9f923-8951-4f90-becb-fc85ff30a41b-istiod-ca-cert\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703066    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703100    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-data\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703156    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znrr5\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-kube-api-access-znrr5\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703199    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-token\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703240    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-envoy\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703341 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703305    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-certs\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703563 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703354    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-credential-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.703902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703875    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-credential-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.704021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703938    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-socket\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.704021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.703937    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e9f923-8951-4f90-becb-fc85ff30a41b-istiod-ca-cert\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.704235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.704212    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-data\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.704473 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.704449    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-certs\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.706889 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.706859    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-envoy\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.707109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.707075    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-podinfo\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.723214 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.723175    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-token\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.723393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.723329    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znrr5\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-kube-api-access-znrr5\") pod \"e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") " pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:47.864584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.864535    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"]
Oct 30 13:48:47.896534 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:47.896502    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:48.258754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.258721    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"]
Oct 30 13:48:48.259849 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:48:48.259821    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8e9f923_8951_4f90_becb_fc85ff30a41b.slice/crio-26ed2fb2a662d4402fd1bbae194236a146909cc347373c4cfa527ed3c2460723 WatchSource:0}: Error finding container 26ed2fb2a662d4402fd1bbae194236a146909cc347373c4cfa527ed3c2460723: Status 404 returned error can't find the container with id 26ed2fb2a662d4402fd1bbae194236a146909cc347373c4cfa527ed3c2460723
Oct 30 13:48:48.262893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.262860    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:48.262966 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.262933    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:48.263007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.262968    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:48:48.426983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.426834    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" event={"ID":"47f9a857-ba8b-4c06-bd76-faa29e505dbf","Type":"ContainerStarted","Data":"2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8"}
Oct 30 13:48:48.426983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.426914    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" event={"ID":"47f9a857-ba8b-4c06-bd76-faa29e505dbf","Type":"ContainerStarted","Data":"0cea6d1b33f2d4d73e6830031918f47a1824c64413960489e411cb979be25742"}
Oct 30 13:48:48.426983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.426949    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:48:48.429253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.429113    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" event={"ID":"a8e9f923-8951-4f90-becb-fc85ff30a41b","Type":"ContainerStarted","Data":"f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71"}
Oct 30 13:48:48.429253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.429171    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" event={"ID":"a8e9f923-8951-4f90-becb-fc85ff30a41b","Type":"ContainerStarted","Data":"26ed2fb2a662d4402fd1bbae194236a146909cc347373c4cfa527ed3c2460723"}
Oct 30 13:48:48.477696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.477622    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" podStartSLOduration=1.47760597 podStartE2EDuration="1.47760597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:48.475008643 +0000 UTC m=+2738.735246878" watchObservedRunningTime="2026-10-30 13:48:48.47760597 +0000 UTC m=+2738.737844233"
Oct 30 13:48:48.521123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.521024    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" podStartSLOduration=1.521007774 podStartE2EDuration="1.521007774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:48:48.517093628 +0000 UTC m=+2738.777331866" watchObservedRunningTime="2026-10-30 13:48:48.521007774 +0000 UTC m=+2738.781246106"
Oct 30 13:48:48.897083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:48.897031    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:49.043053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.043018    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"]
Oct 30 13:48:49.047370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.047342    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"]
Oct 30 13:48:49.060950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.060910    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfb52394-cjt44"]
Oct 30 13:48:49.068252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.068211    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e4d88c78-7s44v"]
Oct 30 13:48:49.885194 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.885158    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a6d1b0f-2965-41ec-a32e-4a688d8542ef" path="/var/lib/kubelet/pods/8a6d1b0f-2965-41ec-a32e-4a688d8542ef/volumes"
Oct 30 13:48:49.885766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.885750    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e" path="/var/lib/kubelet/pods/fa0e25fa-ae47-4b5e-8df3-78e61c9d2e7e/volumes"
Oct 30 13:48:49.899553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.899519    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:48:49.899790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:49.899577    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 13:48:50.897020 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:50.896983    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:48:50.897457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:50.897055    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 13:48:51.897515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:51.897471    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:48:51.898205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:51.897535    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 13:48:52.898103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:52.898017    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:48:52.898103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:52.898080    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 13:48:53.897002 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:53.896959    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused" start-of-body=
Oct 30 13:48:53.897217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:53.897026    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.18:15021/healthz/ready\": dial tcp 10.133.1.18:15021: connect: connection refused"
Oct 30 13:48:54.903173 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:54.903137    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:54.903663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:54.903608    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:54.904344 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:54.904327    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:48:59.439997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:48:59.439965    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:49:01.036103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:01.036058    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"]
Oct 30 13:49:01.040056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:01.040024    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4d6bbd5f-kswrs"]
Oct 30 13:49:01.882656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:01.882598    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="040eee6b-0b4b-4651-b56f-81edd16b4e2c" path="/var/lib/kubelet/pods/040eee6b-0b4b-4651-b56f-81edd16b4e2c/volumes"
Oct 30 13:49:03.039753 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:03.039715    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"]
Oct 30 13:49:03.051279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:03.051247    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-72d40826-rc45p"]
Oct 30 13:49:03.880275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:03.880239    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9a2737-debf-434c-9eda-7408d24906f3" path="/var/lib/kubelet/pods/9e9a2737-debf-434c-9eda-7408d24906f3/volumes"
Oct 30 13:49:04.044127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:04.044091    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"]
Oct 30 13:49:04.056589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:04.056557    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c83715b-hfwdq"]
Oct 30 13:49:05.880651 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:05.880593    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619a17f0-c72d-4071-9daf-d15e69524416" path="/var/lib/kubelet/pods/619a17f0-c72d-4071-9daf-d15e69524416/volumes"
Oct 30 13:49:08.211288 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:49:08.211246    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:49:13.068031 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:13.067999    4291 scope.go:122] "RemoveContainer" containerID="3fb79cf00dc57ebb2af59e818b360310b16b95289d8cf9bac6d33e6395347db0"
Oct 30 13:49:15.101222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:15.101163    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"]
Oct 30 13:49:20.102417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.102351    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc" podUID="33aab158-3aa0-4e83-aaf2-54905aef878b" containerName="payload-processing" containerID="cri-o://19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1" gracePeriod=25
Oct 30 13:49:20.400834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.400800    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:49:20.504387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.504349    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-routing-overlay-volume\") pod \"33aab158-3aa0-4e83-aaf2-54905aef878b\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") "
Oct 30 13:49:20.504564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.504415    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-provider-credentials\") pod \"33aab158-3aa0-4e83-aaf2-54905aef878b\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") "
Oct 30 13:49:20.504564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.504457    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-plugins-config-volume\") pod \"33aab158-3aa0-4e83-aaf2-54905aef878b\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") "
Oct 30 13:49:20.504564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.504541    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-kube-api-access-2fd6m\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-kube-api-access-2fd6m\") pod \"33aab158-3aa0-4e83-aaf2-54905aef878b\" (UID: \"33aab158-3aa0-4e83-aaf2-54905aef878b\") "
Oct 30 13:49:20.504862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.504834    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-routing-overlay-volume" pod "33aab158-3aa0-4e83-aaf2-54905aef878b" (UID: "33aab158-3aa0-4e83-aaf2-54905aef878b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:20.504950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.504884    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-plugins-config-volume" pod "33aab158-3aa0-4e83-aaf2-54905aef878b" (UID: "33aab158-3aa0-4e83-aaf2-54905aef878b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:20.505239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.505221    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:20.505293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.505242    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33aab158-3aa0-4e83-aaf2-54905aef878b-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:20.506690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.506664    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-kube-api-access-2fd6m" pod "33aab158-3aa0-4e83-aaf2-54905aef878b" (UID: "33aab158-3aa0-4e83-aaf2-54905aef878b"). InnerVolumeSpecName "kube-api-access-2fd6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:20.506826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.506808    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-provider-credentials" pod "33aab158-3aa0-4e83-aaf2-54905aef878b" (UID: "33aab158-3aa0-4e83-aaf2-54905aef878b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:20.606806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.606768    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fd6m\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-kube-api-access-2fd6m\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:20.606806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.606802    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/33aab158-3aa0-4e83-aaf2-54905aef878b-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:20.782302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.782190    4291 generic.go:356] "Generic (PLEG): container finished" podID="33aab158-3aa0-4e83-aaf2-54905aef878b" containerID="19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1" exitCode=0
Oct 30 13:49:20.782302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.782243    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"
Oct 30 13:49:20.782302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.782270    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc" event={"ID":"33aab158-3aa0-4e83-aaf2-54905aef878b","Type":"ContainerDied","Data":"19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1"}
Oct 30 13:49:20.782579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.782326    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc" event={"ID":"33aab158-3aa0-4e83-aaf2-54905aef878b","Type":"ContainerDied","Data":"b03177ac502b75ad3114a6236269210aa4ef49588fdb39af617b60c7cdb3ab1d"}
Oct 30 13:49:20.782579 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.782349    4291 scope.go:122] "RemoveContainer" containerID="19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1"
Oct 30 13:49:20.794960 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.794935    4291 scope.go:122] "RemoveContainer" containerID="19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1"
Oct 30 13:49:20.795346 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:49:20.795326    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1\": container with ID starting with 19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1 not found: ID does not exist" containerID="19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1"
Oct 30 13:49:20.795399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.795355    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1"} err="failed to get container status \"19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1\": rpc error: code = NotFound desc = could not find container \"19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1\": container with ID starting with 19d46862874b069bbca687a4bcce0292455698e44f178d1aebaab9a40147ffb1 not found: ID does not exist"
Oct 30 13:49:20.835085 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.835043    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"]
Oct 30 13:49:20.838910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:20.838882    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d2cecb/payload-processing-external-model-e2e-extmod-b-d2cecb-8f97p7jzc"]
Oct 30 13:49:21.894658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:21.887842    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33aab158-3aa0-4e83-aaf2-54905aef878b" path="/var/lib/kubelet/pods/33aab158-3aa0-4e83-aaf2-54905aef878b/volumes"
Oct 30 13:49:22.309708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.308691    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"]
Oct 30 13:49:22.309904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.309829    4291 cpu_manager.go:414] "containerMap: removing container" podUID="33aab158-3aa0-4e83-aaf2-54905aef878b" containerName="payload-processing"
Oct 30 13:49:22.309904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.309855    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33aab158-3aa0-4e83-aaf2-54905aef878b" containerName="payload-processing"
Oct 30 13:49:22.313657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.310081    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="33aab158-3aa0-4e83-aaf2-54905aef878b" containerName="payload-processing"
Oct 30 13:49:22.322661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.321423    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.404145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.403935    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:49:22.404145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.403989    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:49:22.404145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.404032    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:49:22.404145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.404098    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:49:22.507785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.507746    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"]
Oct 30 13:49:22.578771 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.578681    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx9mp\" (UniqueName: \"kubernetes.io/projected/fc2681ae-9af3-46d2-a340-cd9b07a8b246-kube-api-access-nx9mp\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.579027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.578864    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.579101 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.579083    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.680990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.680949    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx9mp\" (UniqueName: \"kubernetes.io/projected/fc2681ae-9af3-46d2-a340-cd9b07a8b246-kube-api-access-nx9mp\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.681207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.681049    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.681207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.681190    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.681844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.681816    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.683586 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.683558    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-xks-ca\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.699655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.699614    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx9mp\" (UniqueName: \"kubernetes.io/projected/fc2681ae-9af3-46d2-a340-cd9b07a8b246-kube-api-access-nx9mp\") pod \"maas-api-revoke-sub-96e1e746-59snf\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:22.798645 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:22.798581    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:23.111992 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:49:23.111921    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2681ae_9af3_46d2_a340_cd9b07a8b246.slice/crio-22dcfcb79af2f9a510c4d00c90f22d2c2f63ff11edc843945acd5d51f408e13a WatchSource:0}: Error finding container 22dcfcb79af2f9a510c4d00c90f22d2c2f63ff11edc843945acd5d51f408e13a: Status 404 returned error can't find the container with id 22dcfcb79af2f9a510c4d00c90f22d2c2f63ff11edc843945acd5d51f408e13a
Oct 30 13:49:23.118799 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:23.118769    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"]
Oct 30 13:49:23.810231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:23.810127    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf" event={"ID":"fc2681ae-9af3-46d2-a340-cd9b07a8b246","Type":"ContainerStarted","Data":"ddf9115ddf2f0fb4a83b6eff2037de4de680488b4dca1741c146b6a9e5834339"}
Oct 30 13:49:23.810231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:23.810197    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf" event={"ID":"fc2681ae-9af3-46d2-a340-cd9b07a8b246","Type":"ContainerStarted","Data":"22dcfcb79af2f9a510c4d00c90f22d2c2f63ff11edc843945acd5d51f408e13a"}
Oct 30 13:49:23.852805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:23.852738    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf" podStartSLOduration=1.8527163469999999 podStartE2EDuration="1.852716347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:23.849480363 +0000 UTC m=+2774.109718598" watchObservedRunningTime="2026-10-30 13:49:23.852716347 +0000 UTC m=+2774.112954583"
Oct 30 13:49:24.819378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:24.819339    4291 generic.go:356] "Generic (PLEG): container finished" podID="fc2681ae-9af3-46d2-a340-cd9b07a8b246" containerID="ddf9115ddf2f0fb4a83b6eff2037de4de680488b4dca1741c146b6a9e5834339" exitCode=0
Oct 30 13:49:24.819378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:24.819370    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf" event={"ID":"fc2681ae-9af3-46d2-a340-cd9b07a8b246","Type":"ContainerDied","Data":"ddf9115ddf2f0fb4a83b6eff2037de4de680488b4dca1741c146b6a9e5834339"}
Oct 30 13:49:26.007361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.007333    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:26.026938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.026908    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-xks-ca\") pod \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") "
Oct 30 13:49:26.027139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.027002    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-openshift-ca\") pod \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") "
Oct 30 13:49:26.027139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.027118    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc2681ae-9af3-46d2-a340-cd9b07a8b246-kube-api-access-nx9mp\" (UniqueName: \"kubernetes.io/projected/fc2681ae-9af3-46d2-a340-cd9b07a8b246-kube-api-access-nx9mp\") pod \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\" (UID: \"fc2681ae-9af3-46d2-a340-cd9b07a8b246\") "
Oct 30 13:49:26.027390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.027360    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-openshift-ca" pod "fc2681ae-9af3-46d2-a340-cd9b07a8b246" (UID: "fc2681ae-9af3-46d2-a340-cd9b07a8b246"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:26.027775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.027757    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:26.029249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.029226    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc2681ae-9af3-46d2-a340-cd9b07a8b246-kube-api-access-nx9mp" pod "fc2681ae-9af3-46d2-a340-cd9b07a8b246" (UID: "fc2681ae-9af3-46d2-a340-cd9b07a8b246"). InnerVolumeSpecName "kube-api-access-nx9mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:26.029345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.029262    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-xks-ca" pod "fc2681ae-9af3-46d2-a340-cd9b07a8b246" (UID: "fc2681ae-9af3-46d2-a340-cd9b07a8b246"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:49:26.129196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.129160    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc2681ae-9af3-46d2-a340-cd9b07a8b246-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:26.129196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.129194    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx9mp\" (UniqueName: \"kubernetes.io/projected/fc2681ae-9af3-46d2-a340-cd9b07a8b246-kube-api-access-nx9mp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:26.836729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.836694    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"
Oct 30 13:49:26.836937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.836693    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf" event={"ID":"fc2681ae-9af3-46d2-a340-cd9b07a8b246","Type":"ContainerDied","Data":"22dcfcb79af2f9a510c4d00c90f22d2c2f63ff11edc843945acd5d51f408e13a"}
Oct 30 13:49:26.836937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:26.836875    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22dcfcb79af2f9a510c4d00c90f22d2c2f63ff11edc843945acd5d51f408e13a"
Oct 30 13:49:30.158500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:30.158458    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:49:37.210602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.210566    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"]
Oct 30 13:49:37.211427 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.211408    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fc2681ae-9af3-46d2-a340-cd9b07a8b246" containerName="revoke-keys"
Oct 30 13:49:37.211494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.211428    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc2681ae-9af3-46d2-a340-cd9b07a8b246" containerName="revoke-keys"
Oct 30 13:49:37.211578 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.211568    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc2681ae-9af3-46d2-a340-cd9b07a8b246" containerName="revoke-keys"
Oct 30 13:49:37.215930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.215912    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.231786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.231755    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"]
Oct 30 13:49:37.239603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.239568    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-e2e--175bf124-dockercfg-k2xvk\""
Oct 30 13:49:37.239862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.239591    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"agc-cred-80995a0f88ba\""
Oct 30 13:49:37.239862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.239613    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"agc-cred-df536a73b1ac\""
Oct 30 13:49:37.239862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.239703    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"routing-overlay\""
Oct 30 13:49:37.239862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.239788    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"openshift-service-ca.crt\""
Oct 30 13:49:37.242471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.242429    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"payload-processing-external-model-plugins-e2e-extmod-a-310546\""
Oct 30 13:49:37.242628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.242537    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-310546\"/\"kube-root-ca.crt\""
Oct 30 13:49:37.330188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.330150    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.330403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.330382    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m92zh\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-kube-api-access-m92zh\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.330501 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.330445    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.330543 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.330502    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.431739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.431701    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m92zh\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-kube-api-access-m92zh\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.431932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.431794    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.431932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.431821    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.431932 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.431917    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.432538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.432509    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.432672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.432556    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.434371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.434346    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.452478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.452441    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m92zh\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-kube-api-access-m92zh\") pod \"payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") " pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.535926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.535828    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:37.940567 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.940539    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"]
Oct 30 13:49:37.942931 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:49:37.942906    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9dee9de_66e9_400d_abfb_8eb37e153951.slice/crio-a98eaff463e89ee3c0818fd560f166a9207b659bb94c2a5718059e361201b50c WatchSource:0}: Error finding container a98eaff463e89ee3c0818fd560f166a9207b659bb94c2a5718059e361201b50c: Status 404 returned error can't find the container with id a98eaff463e89ee3c0818fd560f166a9207b659bb94c2a5718059e361201b50c
Oct 30 13:49:37.973709 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:37.973669    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc" event={"ID":"b9dee9de-66e9-400d-abfb-8eb37e153951","Type":"ContainerStarted","Data":"a98eaff463e89ee3c0818fd560f166a9207b659bb94c2a5718059e361201b50c"}
Oct 30 13:49:38.212040 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:49:38.211946    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:49:38.982568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:38.982532    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc" event={"ID":"b9dee9de-66e9-400d-abfb-8eb37e153951","Type":"ContainerStarted","Data":"d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041"}
Oct 30 13:49:38.982819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:38.982626    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:39.021951 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:39.021894    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc" podStartSLOduration=2.021879251 podStartE2EDuration="2.021879251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:39.020503318 +0000 UTC m=+2789.280741552" watchObservedRunningTime="2026-10-30 13:49:39.021879251 +0000 UTC m=+2789.282117485"
Oct 30 13:49:42.038575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:42.038537    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"]
Oct 30 13:49:42.070512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:42.070455    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2d74f52f-jg4pb"]
Oct 30 13:49:43.676327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.676204    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"]
Oct 30 13:49:43.710453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.710407    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.727016 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.726951    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:49:43.727201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.727033    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"]
Oct 30 13:49:43.735145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.734938    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:49:43.735145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.735014    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:49:43.735145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.735111    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:49:43.763322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.757081    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.763322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.757180    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.763322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.757242    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-df9rm\" (UniqueName: \"kubernetes.io/projected/0ddab16a-250d-4d79-98c0-75a9828a7604-kube-api-access-df9rm\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.858259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.858202    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.858461 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.858269    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-df9rm\" (UniqueName: \"kubernetes.io/projected/0ddab16a-250d-4d79-98c0-75a9828a7604-kube-api-access-df9rm\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.858461 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.858416    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.858937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.858910    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.860929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.860906    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.881126 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.881091    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-df9rm\" (UniqueName: \"kubernetes.io/projected/0ddab16a-250d-4d79-98c0-75a9828a7604-kube-api-access-df9rm\") pod \"maas-api-revoke-sub-a1704930-j7tjn\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:43.882452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:43.882424    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38d8474a-3aa9-4490-9f48-42af329743b7" path="/var/lib/kubelet/pods/38d8474a-3aa9-4490-9f48-42af329743b7/volumes"
Oct 30 13:49:44.090714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:44.090323    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:44.234156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:44.234118    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"]
Oct 30 13:49:44.238061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:44.238026    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:44.504830 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:44.503972    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"]
Oct 30 13:49:44.512171 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:49:44.512134    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddab16a_250d_4d79_98c0_75a9828a7604.slice/crio-28af22569d495ac0b15c58636d7fb910bfa4583b30a14c02cdf65fb553502613 WatchSource:0}: Error finding container 28af22569d495ac0b15c58636d7fb910bfa4583b30a14c02cdf65fb553502613: Status 404 returned error can't find the container with id 28af22569d495ac0b15c58636d7fb910bfa4583b30a14c02cdf65fb553502613
Oct 30 13:49:45.083615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:45.083577    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn" event={"ID":"0ddab16a-250d-4d79-98c0-75a9828a7604","Type":"ContainerStarted","Data":"3286737e3d9e9a2f312cdd46d90b64d3f28dbc12971c5ce9ba6c4ba28e3d6b48"}
Oct 30 13:49:45.084092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:45.083679    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn" event={"ID":"0ddab16a-250d-4d79-98c0-75a9828a7604","Type":"ContainerStarted","Data":"28af22569d495ac0b15c58636d7fb910bfa4583b30a14c02cdf65fb553502613"}
Oct 30 13:49:45.123711 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:45.122527    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn" podStartSLOduration=2.122504749 podStartE2EDuration="2.122504749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:45.120550999 +0000 UTC m=+2795.380789234" watchObservedRunningTime="2026-10-30 13:49:45.122504749 +0000 UTC m=+2795.382742988"
Oct 30 13:49:46.092301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:46.092268    4291 generic.go:356] "Generic (PLEG): container finished" podID="0ddab16a-250d-4d79-98c0-75a9828a7604" containerID="3286737e3d9e9a2f312cdd46d90b64d3f28dbc12971c5ce9ba6c4ba28e3d6b48" exitCode=0
Oct 30 13:49:46.092737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:46.092359    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn" event={"ID":"0ddab16a-250d-4d79-98c0-75a9828a7604","Type":"ContainerDied","Data":"3286737e3d9e9a2f312cdd46d90b64d3f28dbc12971c5ce9ba6c4ba28e3d6b48"}
Oct 30 13:49:47.273182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.273155    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:47.342196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.342154    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ddab16a-250d-4d79-98c0-75a9828a7604-kube-api-access-df9rm\" (UniqueName: \"kubernetes.io/projected/0ddab16a-250d-4d79-98c0-75a9828a7604-kube-api-access-df9rm\") pod \"0ddab16a-250d-4d79-98c0-75a9828a7604\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") "
Oct 30 13:49:47.342383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.342222    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-openshift-ca\") pod \"0ddab16a-250d-4d79-98c0-75a9828a7604\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") "
Oct 30 13:49:47.342472 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.342455    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-xks-ca\") pod \"0ddab16a-250d-4d79-98c0-75a9828a7604\" (UID: \"0ddab16a-250d-4d79-98c0-75a9828a7604\") "
Oct 30 13:49:47.342597 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.342569    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-openshift-ca" pod "0ddab16a-250d-4d79-98c0-75a9828a7604" (UID: "0ddab16a-250d-4d79-98c0-75a9828a7604"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:47.343015 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.342991    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:47.344664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.344594    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-xks-ca" pod "0ddab16a-250d-4d79-98c0-75a9828a7604" (UID: "0ddab16a-250d-4d79-98c0-75a9828a7604"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:49:47.344997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.344974    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ddab16a-250d-4d79-98c0-75a9828a7604-kube-api-access-df9rm" pod "0ddab16a-250d-4d79-98c0-75a9828a7604" (UID: "0ddab16a-250d-4d79-98c0-75a9828a7604"). InnerVolumeSpecName "kube-api-access-df9rm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:47.444422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.444384    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-df9rm\" (UniqueName: \"kubernetes.io/projected/0ddab16a-250d-4d79-98c0-75a9828a7604-kube-api-access-df9rm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:47.444422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:47.444417    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ddab16a-250d-4d79-98c0-75a9828a7604-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:48.118197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:48.118162    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"
Oct 30 13:49:48.118382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:48.118198    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn" event={"ID":"0ddab16a-250d-4d79-98c0-75a9828a7604","Type":"ContainerDied","Data":"28af22569d495ac0b15c58636d7fb910bfa4583b30a14c02cdf65fb553502613"}
Oct 30 13:49:48.118382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:48.118280    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28af22569d495ac0b15c58636d7fb910bfa4583b30a14c02cdf65fb553502613"
Oct 30 13:49:49.235096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.235046    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc" podUID="b9dee9de-66e9-400d-abfb-8eb37e153951" containerName="payload-processing" containerID="cri-o://d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041" gracePeriod=25
Oct 30 13:49:49.740943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.740915    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:49.814238 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.814187    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-plugins-config-volume\") pod \"b9dee9de-66e9-400d-abfb-8eb37e153951\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") "
Oct 30 13:49:49.814447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.814307    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-routing-overlay-volume\") pod \"b9dee9de-66e9-400d-abfb-8eb37e153951\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") "
Oct 30 13:49:49.814447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.814377    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-provider-credentials\") pod \"b9dee9de-66e9-400d-abfb-8eb37e153951\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") "
Oct 30 13:49:49.814447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.814429    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-kube-api-access-m92zh\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-kube-api-access-m92zh\") pod \"b9dee9de-66e9-400d-abfb-8eb37e153951\" (UID: \"b9dee9de-66e9-400d-abfb-8eb37e153951\") "
Oct 30 13:49:49.814730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.814700    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-routing-overlay-volume" pod "b9dee9de-66e9-400d-abfb-8eb37e153951" (UID: "b9dee9de-66e9-400d-abfb-8eb37e153951"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:49.814840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.814712    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-plugins-config-volume" pod "b9dee9de-66e9-400d-abfb-8eb37e153951" (UID: "b9dee9de-66e9-400d-abfb-8eb37e153951"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:49.815135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.815113    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:49.815228 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.815138    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/b9dee9de-66e9-400d-abfb-8eb37e153951-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:49.817328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.817299    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-provider-credentials" pod "b9dee9de-66e9-400d-abfb-8eb37e153951" (UID: "b9dee9de-66e9-400d-abfb-8eb37e153951"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:49.817460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.817344    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-kube-api-access-m92zh" pod "b9dee9de-66e9-400d-abfb-8eb37e153951" (UID: "b9dee9de-66e9-400d-abfb-8eb37e153951"). InnerVolumeSpecName "kube-api-access-m92zh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:49.916849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.916812    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:49.916849 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:49.916843    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m92zh\" (UniqueName: \"kubernetes.io/projected/b9dee9de-66e9-400d-abfb-8eb37e153951-kube-api-access-m92zh\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:50.135720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.135684    4291 generic.go:356] "Generic (PLEG): container finished" podID="b9dee9de-66e9-400d-abfb-8eb37e153951" containerID="d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041" exitCode=0
Oct 30 13:49:50.135904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.135734    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"
Oct 30 13:49:50.135904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.135773    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc" event={"ID":"b9dee9de-66e9-400d-abfb-8eb37e153951","Type":"ContainerDied","Data":"d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041"}
Oct 30 13:49:50.135904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.135830    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc" event={"ID":"b9dee9de-66e9-400d-abfb-8eb37e153951","Type":"ContainerDied","Data":"a98eaff463e89ee3c0818fd560f166a9207b659bb94c2a5718059e361201b50c"}
Oct 30 13:49:50.135904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.135860    4291 scope.go:122] "RemoveContainer" containerID="d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041"
Oct 30 13:49:50.147743 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.147714    4291 scope.go:122] "RemoveContainer" containerID="d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041"
Oct 30 13:49:50.148158 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:49:50.148100    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041\": container with ID starting with d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041 not found: ID does not exist" containerID="d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041"
Oct 30 13:49:50.148158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.148141    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041"} err="failed to get container status \"d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041\": rpc error: code = NotFound desc = could not find container \"d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041\": container with ID starting with d356464ac18a5218f0d696ef56d4e02fe94b1a9f021485160ae34fa52da47041 not found: ID does not exist"
Oct 30 13:49:50.189240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.189205    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"]
Oct 30 13:49:50.200990 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.200953    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-310546/payload-processing-external-model-e2e-extmod-a-310546-6bddtjxlc"]
Oct 30 13:49:50.478422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.478330    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"]
Oct 30 13:49:50.479057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.479039    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0ddab16a-250d-4d79-98c0-75a9828a7604" containerName="revoke-keys"
Oct 30 13:49:50.479117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.479058    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ddab16a-250d-4d79-98c0-75a9828a7604" containerName="revoke-keys"
Oct 30 13:49:50.479117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.479088    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b9dee9de-66e9-400d-abfb-8eb37e153951" containerName="payload-processing"
Oct 30 13:49:50.479117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.479093    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9dee9de-66e9-400d-abfb-8eb37e153951" containerName="payload-processing"
Oct 30 13:49:50.479226 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.479216    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ddab16a-250d-4d79-98c0-75a9828a7604" containerName="revoke-keys"
Oct 30 13:49:50.479266 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.479233    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9dee9de-66e9-400d-abfb-8eb37e153951" containerName="payload-processing"
Oct 30 13:49:50.484292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.484266    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.500127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.500090    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"]
Oct 30 13:49:50.500629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.500607    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"agc-cred-9af8e5e90455\""
Oct 30 13:49:50.500829 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.500811    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"kube-root-ca.crt\""
Oct 30 13:49:50.507897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.507768    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"openshift-service-ca.crt\""
Oct 30 13:49:50.507897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.507802    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"payload-processing-external-model-plugins-e2e-extmod-a-bd2fed\""
Oct 30 13:49:50.507897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.507830    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"routing-overlay\""
Oct 30 13:49:50.507897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.507838    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-bd2fed\"/\"payload-processing-external-model-e2e--fb32b92e-dockercfg-qtdwq\""
Oct 30 13:49:50.530140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.530097    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-627f9\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-kube-api-access-627f9\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.530140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.530143    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.530452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.530237    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.530517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.530468    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.649663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.648699    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.649663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.648829    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-627f9\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-kube-api-access-627f9\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.649663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.648858    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.649663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.648906    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.656866 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.656831    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.695986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.666091    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.695986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.674322    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.695986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.694487    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-627f9\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-kube-api-access-627f9\") pod \"payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") " pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:50.800119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:50.800015    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:51.112559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:51.112518    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"]
Oct 30 13:49:51.145199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:51.145173    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd" event={"ID":"894e3f5c-a5d0-4960-87b2-15a7dd1078b7","Type":"ContainerStarted","Data":"f2d6c9e83314a00465703c9f9bf27b144c8a55b1aed0260dae01df85aa3776ea"}
Oct 30 13:49:51.881863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:51.881821    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9dee9de-66e9-400d-abfb-8eb37e153951" path="/var/lib/kubelet/pods/b9dee9de-66e9-400d-abfb-8eb37e153951/volumes"
Oct 30 13:49:52.155668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:52.155556    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd" event={"ID":"894e3f5c-a5d0-4960-87b2-15a7dd1078b7","Type":"ContainerStarted","Data":"0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46"}
Oct 30 13:49:52.155668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:52.155657    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:49:52.194702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:52.194631    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd" podStartSLOduration=2.194617381 podStartE2EDuration="2.194617381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:49:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:49:52.193287685 +0000 UTC m=+2802.453525918" watchObservedRunningTime="2026-10-30 13:49:52.194617381 +0000 UTC m=+2802.454855615"
Oct 30 13:49:54.076965 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.076929    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"]
Oct 30 13:49:54.089215 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.089185    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a40bf410-5w4xv"]
Oct 30 13:49:54.532260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.532227    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"]
Oct 30 13:49:54.537814 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.537792    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.553755 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.553728    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"]
Oct 30 13:49:54.560478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.560451    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:49:54.560690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.560474    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:49:54.560806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.560483    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:49:54.561108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.561081    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:49:54.625533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.625496    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.625774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.625618    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxq8m\" (UniqueName: \"kubernetes.io/projected/06ee0849-4bdc-466b-ac19-fbecb61152a2-kube-api-access-vxq8m\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.625859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.625788    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.727750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.727713    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.727938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.727784    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxq8m\" (UniqueName: \"kubernetes.io/projected/06ee0849-4bdc-466b-ac19-fbecb61152a2-kube-api-access-vxq8m\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.727938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.727827    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.728469 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.728446    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.730261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.730237    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.746727 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.746695    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxq8m\" (UniqueName: \"kubernetes.io/projected/06ee0849-4bdc-466b-ac19-fbecb61152a2-kube-api-access-vxq8m\") pod \"maas-api-revoke-sub-31af5609-xvncb\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:54.854778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:54.854752    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:55.165558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:55.165529    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"]
Oct 30 13:49:55.166766 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:49:55.166744    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ee0849_4bdc_466b_ac19_fbecb61152a2.slice/crio-1e479b6aca397698b1529fbd83db5ce0e9c8af66e923dc05f426e50c0adf8a6a WatchSource:0}: Error finding container 1e479b6aca397698b1529fbd83db5ce0e9c8af66e923dc05f426e50c0adf8a6a: Status 404 returned error can't find the container with id 1e479b6aca397698b1529fbd83db5ce0e9c8af66e923dc05f426e50c0adf8a6a
Oct 30 13:49:55.179325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:55.179296    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb" event={"ID":"06ee0849-4bdc-466b-ac19-fbecb61152a2","Type":"ContainerStarted","Data":"1e479b6aca397698b1529fbd83db5ce0e9c8af66e923dc05f426e50c0adf8a6a"}
Oct 30 13:49:55.881523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:55.881484    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72412330-2213-4c10-9d32-8860d697c9eb" path="/var/lib/kubelet/pods/72412330-2213-4c10-9d32-8860d697c9eb/volumes"
Oct 30 13:49:56.187924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:56.187840    4291 generic.go:356] "Generic (PLEG): container finished" podID="06ee0849-4bdc-466b-ac19-fbecb61152a2" containerID="57a14e8f7d6ddff335e0d91000bbfb00bcc8c355842ca8fd414933e75b87e2c3" exitCode=0
Oct 30 13:49:56.188304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:56.187929    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb" event={"ID":"06ee0849-4bdc-466b-ac19-fbecb61152a2","Type":"ContainerDied","Data":"57a14e8f7d6ddff335e0d91000bbfb00bcc8c355842ca8fd414933e75b87e2c3"}
Oct 30 13:49:57.382026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.382001    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:57.478062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.477976    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-xks-ca\") pod \"06ee0849-4bdc-466b-ac19-fbecb61152a2\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") "
Oct 30 13:49:57.478062 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.478026    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ee0849-4bdc-466b-ac19-fbecb61152a2-kube-api-access-vxq8m\" (UniqueName: \"kubernetes.io/projected/06ee0849-4bdc-466b-ac19-fbecb61152a2-kube-api-access-vxq8m\") pod \"06ee0849-4bdc-466b-ac19-fbecb61152a2\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") "
Oct 30 13:49:57.478258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.478197    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-openshift-ca\") pod \"06ee0849-4bdc-466b-ac19-fbecb61152a2\" (UID: \"06ee0849-4bdc-466b-ac19-fbecb61152a2\") "
Oct 30 13:49:57.478555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.478529    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-openshift-ca" pod "06ee0849-4bdc-466b-ac19-fbecb61152a2" (UID: "06ee0849-4bdc-466b-ac19-fbecb61152a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:49:57.478937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.478918    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:57.480222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.480193    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-xks-ca" pod "06ee0849-4bdc-466b-ac19-fbecb61152a2" (UID: "06ee0849-4bdc-466b-ac19-fbecb61152a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:49:57.480323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.480221    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ee0849-4bdc-466b-ac19-fbecb61152a2-kube-api-access-vxq8m" pod "06ee0849-4bdc-466b-ac19-fbecb61152a2" (UID: "06ee0849-4bdc-466b-ac19-fbecb61152a2"). InnerVolumeSpecName "kube-api-access-vxq8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:49:57.580322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.580285    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ee0849-4bdc-466b-ac19-fbecb61152a2-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:57.580322 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:57.580315    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxq8m\" (UniqueName: \"kubernetes.io/projected/06ee0849-4bdc-466b-ac19-fbecb61152a2-kube-api-access-vxq8m\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:49:58.206738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:58.206602    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"
Oct 30 13:49:58.206738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:58.206623    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb" event={"ID":"06ee0849-4bdc-466b-ac19-fbecb61152a2","Type":"ContainerDied","Data":"1e479b6aca397698b1529fbd83db5ce0e9c8af66e923dc05f426e50c0adf8a6a"}
Oct 30 13:49:58.206738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:49:58.206710    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e479b6aca397698b1529fbd83db5ce0e9c8af66e923dc05f426e50c0adf8a6a"
Oct 30 13:50:03.167422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:03.167328    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:50:08.211984 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:50:08.211946    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:50:10.611662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:10.610748    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:50:13.112855 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:13.112825    4291 scope.go:122] "RemoveContainer" containerID="9e3d40c8b99aa8ab16b06a360e794732568492b0fa1d781857ef054205f37c99"
Oct 30 13:50:13.123527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:13.123508    4291 scope.go:122] "RemoveContainer" containerID="bd655b0dfc0a59413e42fc4c470f7651184f59f0485f87efad529defd1b61389"
Oct 30 13:50:13.133500 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:13.133477    4291 scope.go:122] "RemoveContainer" containerID="9aa84cc4e0ccbeef8ceb30777cd447216b3eca8297a5d5114a7a433f161d6dd7"
Oct 30 13:50:13.142823 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:13.142773    4291 scope.go:122] "RemoveContainer" containerID="667874c1904adfba568a707c57add78f323de20bc98518da4cfa92cd117149e1"
Oct 30 13:50:13.151456 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:13.151436    4291 scope.go:122] "RemoveContainer" containerID="f4de12dcd85b610f272aa0a049c689c503cbb87dd55dcc2c621086e5b4928f7a"
Oct 30 13:50:13.160250 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:13.160230    4291 scope.go:122] "RemoveContainer" containerID="4b630079fac98b91975839bf087f5a728cc79a82cdf24b461583997e48aa7172"
Oct 30 13:50:13.169557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:13.169541    4291 scope.go:122] "RemoveContainer" containerID="fb0d3bffec6af36006b60961a2943f963da1e39215aba465be6801c326a66622"
Oct 30 13:50:14.254927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.254889    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"]
Oct 30 13:50:14.256003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.255976    4291 cpu_manager.go:414] "containerMap: removing container" podUID="06ee0849-4bdc-466b-ac19-fbecb61152a2" containerName="revoke-keys"
Oct 30 13:50:14.256094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.256005    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ee0849-4bdc-466b-ac19-fbecb61152a2" containerName="revoke-keys"
Oct 30 13:50:14.256207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.256194    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ee0849-4bdc-466b-ac19-fbecb61152a2" containerName="revoke-keys"
Oct 30 13:50:14.262548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.262523    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.275343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.275310    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.275506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.275480    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.275794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.275769    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw66h\" (UniqueName: \"kubernetes.io/projected/497a939d-f8d4-4607-8758-c0c3e3c116ca-kube-api-access-zw66h\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.280483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.280453    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"]
Oct 30 13:50:14.282812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.282785    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:50:14.283295 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.283099    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:50:14.283442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.283156    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:50:14.289039 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.289007    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:50:14.382804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.377780    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw66h\" (UniqueName: \"kubernetes.io/projected/497a939d-f8d4-4607-8758-c0c3e3c116ca-kube-api-access-zw66h\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.382804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.377928    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.382804 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.378103    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.386665 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.384073    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.402944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.397102    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.410737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.410696    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw66h\" (UniqueName: \"kubernetes.io/projected/497a939d-f8d4-4607-8758-c0c3e3c116ca-kube-api-access-zw66h\") pod \"maas-api-revoke-sub-60e2635a-mrxsv\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.587644 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.587545    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:14.923018 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:14.922978    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod497a939d_f8d4_4607_8758_c0c3e3c116ca.slice/crio-c977de41222b5b339bfe536b50f3c35e10cd1df9d79c2ada165b9234b2da8dba WatchSource:0}: Error finding container c977de41222b5b339bfe536b50f3c35e10cd1df9d79c2ada165b9234b2da8dba: Status 404 returned error can't find the container with id c977de41222b5b339bfe536b50f3c35e10cd1df9d79c2ada165b9234b2da8dba
Oct 30 13:50:14.931239 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:14.931210    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"]
Oct 30 13:50:15.524661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:15.522489    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv" event={"ID":"497a939d-f8d4-4607-8758-c0c3e3c116ca","Type":"ContainerStarted","Data":"c977de41222b5b339bfe536b50f3c35e10cd1df9d79c2ada165b9234b2da8dba"}
Oct 30 13:50:16.463178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.463136    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"]
Oct 30 13:50:16.468894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.468864    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.474499 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.474466    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"]
Oct 30 13:50:16.484091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.484051    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:16.485361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.484980    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"]
Oct 30 13:50:16.485517 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.485497    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-996b8b\""
Oct 30 13:50:16.498082 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.498043    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"]
Oct 30 13:50:16.499125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.499092    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-996b8b-dockercfg-5xds5\""
Oct 30 13:50:16.520119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.520087    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c87nm\" (UniqueName: \"kubernetes.io/projected/03b01c81-164a-46a8-8a40-da1c7131c10e-kube-api-access-c87nm\") pod \"payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:16.520325 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.520217    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b01c81-164a-46a8-8a40-da1c7131c10e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:16.520437 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.520398    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bebcb8-9b91-4ee8-ae68-b6457e01f397-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.520624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.520601    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmth5\" (UniqueName: \"kubernetes.io/projected/08bebcb8-9b91-4ee8-ae68-b6457e01f397-kube-api-access-hmth5\") pod \"payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.533926 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.533888    4291 generic.go:356] "Generic (PLEG): container finished" podID="497a939d-f8d4-4607-8758-c0c3e3c116ca" containerID="f0561a1b2d5991a6d4e2f32d94f5fec0d40ea4af2c3ffd04c04bff6afd4214c0" exitCode=0
Oct 30 13:50:16.534354 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.533941    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv" event={"ID":"497a939d-f8d4-4607-8758-c0c3e3c116ca","Type":"ContainerDied","Data":"f0561a1b2d5991a6d4e2f32d94f5fec0d40ea4af2c3ffd04c04bff6afd4214c0"}
Oct 30 13:50:16.659657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.643795    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bebcb8-9b91-4ee8-ae68-b6457e01f397-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.659657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.643985    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmth5\" (UniqueName: \"kubernetes.io/projected/08bebcb8-9b91-4ee8-ae68-b6457e01f397-kube-api-access-hmth5\") pod \"payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.659657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.644115    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c87nm\" (UniqueName: \"kubernetes.io/projected/03b01c81-164a-46a8-8a40-da1c7131c10e-kube-api-access-c87nm\") pod \"payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:16.659657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.644204    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b01c81-164a-46a8-8a40-da1c7131c10e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:16.659657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.658506    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b01c81-164a-46a8-8a40-da1c7131c10e-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:16.661409 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.660390    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bebcb8-9b91-4ee8-ae68-b6457e01f397-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.693656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.682012    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c87nm\" (UniqueName: \"kubernetes.io/projected/03b01c81-164a-46a8-8a40-da1c7131c10e-kube-api-access-c87nm\") pod \"payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") " pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:16.725712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.724029    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmth5\" (UniqueName: \"kubernetes.io/projected/08bebcb8-9b91-4ee8-ae68-b6457e01f397-kube-api-access-hmth5\") pod \"payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.797298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.797011    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:16.814021 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:16.813981    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:17.281516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.281486    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"]
Oct 30 13:50:17.282198 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:17.282170    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03b01c81_164a_46a8_8a40_da1c7131c10e.slice/crio-0585a56e9a4b19757bc5de5b53f27753a9c6c575021a3fbe2b00667df4ed68bf WatchSource:0}: Error finding container 0585a56e9a4b19757bc5de5b53f27753a9c6c575021a3fbe2b00667df4ed68bf: Status 404 returned error can't find the container with id 0585a56e9a4b19757bc5de5b53f27753a9c6c575021a3fbe2b00667df4ed68bf
Oct 30 13:50:17.295310 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.295278    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"]
Oct 30 13:50:17.298788 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:17.298760    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08bebcb8_9b91_4ee8_ae68_b6457e01f397.slice/crio-e1d365a6b06d6cdd1830b2d6185a5251acc4351c030fa059bbd54c287b562f8b WatchSource:0}: Error finding container e1d365a6b06d6cdd1830b2d6185a5251acc4351c030fa059bbd54c287b562f8b: Status 404 returned error can't find the container with id e1d365a6b06d6cdd1830b2d6185a5251acc4351c030fa059bbd54c287b562f8b
Oct 30 13:50:17.544259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.544155    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" event={"ID":"08bebcb8-9b91-4ee8-ae68-b6457e01f397","Type":"ContainerStarted","Data":"e08fe47be7e0049f88b315f0cd9e0fb31f1d39371c11a42ab07cc24f2220255b"}
Oct 30 13:50:17.544259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.544217    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" event={"ID":"08bebcb8-9b91-4ee8-ae68-b6457e01f397","Type":"ContainerStarted","Data":"e1d365a6b06d6cdd1830b2d6185a5251acc4351c030fa059bbd54c287b562f8b"}
Oct 30 13:50:17.544259 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.544245    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:17.546027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.546000    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" event={"ID":"03b01c81-164a-46a8-8a40-da1c7131c10e","Type":"ContainerStarted","Data":"82d16ece91e5e045d81c2ada06c0bd78977bc52c477c2c280cbdb882b6d2ab86"}
Oct 30 13:50:17.546159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.546048    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" event={"ID":"03b01c81-164a-46a8-8a40-da1c7131c10e","Type":"ContainerStarted","Data":"0585a56e9a4b19757bc5de5b53f27753a9c6c575021a3fbe2b00667df4ed68bf"}
Oct 30 13:50:17.546159 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.546147    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:17.625465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.625403    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" podStartSLOduration=1.62538216 podStartE2EDuration="1.62538216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:17.623075982 +0000 UTC m=+2827.883314217" watchObservedRunningTime="2026-10-30 13:50:17.62538216 +0000 UTC m=+2827.885620395"
Oct 30 13:50:17.626783 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.626260    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" podStartSLOduration=1.626245983 podStartE2EDuration="1.626245983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:17.582887422 +0000 UTC m=+2827.843125669" watchObservedRunningTime="2026-10-30 13:50:17.626245983 +0000 UTC m=+2827.886484218"
Oct 30 13:50:17.731327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.731296    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:17.761917 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.761880    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-xks-ca\") pod \"497a939d-f8d4-4607-8758-c0c3e3c116ca\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") "
Oct 30 13:50:17.762115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.761955    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/497a939d-f8d4-4607-8758-c0c3e3c116ca-kube-api-access-zw66h\" (UniqueName: \"kubernetes.io/projected/497a939d-f8d4-4607-8758-c0c3e3c116ca-kube-api-access-zw66h\") pod \"497a939d-f8d4-4607-8758-c0c3e3c116ca\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") "
Oct 30 13:50:17.762115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.762032    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-openshift-ca\") pod \"497a939d-f8d4-4607-8758-c0c3e3c116ca\" (UID: \"497a939d-f8d4-4607-8758-c0c3e3c116ca\") "
Oct 30 13:50:17.762421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.762391    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-openshift-ca" pod "497a939d-f8d4-4607-8758-c0c3e3c116ca" (UID: "497a939d-f8d4-4607-8758-c0c3e3c116ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:17.764202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.764165    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-xks-ca" pod "497a939d-f8d4-4607-8758-c0c3e3c116ca" (UID: "497a939d-f8d4-4607-8758-c0c3e3c116ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:17.764380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.764356    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/497a939d-f8d4-4607-8758-c0c3e3c116ca-kube-api-access-zw66h" pod "497a939d-f8d4-4607-8758-c0c3e3c116ca" (UID: "497a939d-f8d4-4607-8758-c0c3e3c116ca"). InnerVolumeSpecName "kube-api-access-zw66h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:17.863784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.863747    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:17.863784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.863781    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zw66h\" (UniqueName: \"kubernetes.io/projected/497a939d-f8d4-4607-8758-c0c3e3c116ca-kube-api-access-zw66h\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:17.863784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:17.863790    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/497a939d-f8d4-4607-8758-c0c3e3c116ca-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:18.555140 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:18.555109    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"
Oct 30 13:50:18.555581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:18.555136    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv" event={"ID":"497a939d-f8d4-4607-8758-c0c3e3c116ca","Type":"ContainerDied","Data":"c977de41222b5b339bfe536b50f3c35e10cd1df9d79c2ada165b9234b2da8dba"}
Oct 30 13:50:18.555581 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:18.555186    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c977de41222b5b339bfe536b50f3c35e10cd1df9d79c2ada165b9234b2da8dba"
Oct 30 13:50:21.025481 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.025447    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"]
Oct 30 13:50:21.026141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.026127    4291 cpu_manager.go:414] "containerMap: removing container" podUID="497a939d-f8d4-4607-8758-c0c3e3c116ca" containerName="revoke-keys"
Oct 30 13:50:21.026233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.026142    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="497a939d-f8d4-4607-8758-c0c3e3c116ca" containerName="revoke-keys"
Oct 30 13:50:21.026279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.026258    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="497a939d-f8d4-4607-8758-c0c3e3c116ca" containerName="revoke-keys"
Oct 30 13:50:21.030107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.030088    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.046480 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.046454    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"]
Oct 30 13:50:21.053535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.053513    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-08c365\""
Oct 30 13:50:21.105674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.105626    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"]
Oct 30 13:50:21.110171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.110146    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.110797 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.110774    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lrgd\" (UniqueName: \"kubernetes.io/projected/ade31a0f-9282-4bde-86a9-e17e7b41f05b-kube-api-access-2lrgd\") pod \"payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.110870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.110842    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade31a0f-9282-4bde-86a9-e17e7b41f05b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.334941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.334856    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lrgd\" (UniqueName: \"kubernetes.io/projected/ade31a0f-9282-4bde-86a9-e17e7b41f05b-kube-api-access-2lrgd\") pod \"payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.346652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.341104    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade31a0f-9282-4bde-86a9-e17e7b41f05b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.346652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.317324    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-08c365-dockercfg-bz2sx\""
Oct 30 13:50:21.346652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.336746    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"]
Oct 30 13:50:21.346652 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.342409    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade31a0f-9282-4bde-86a9-e17e7b41f05b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.368308 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.368242    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lrgd\" (UniqueName: \"kubernetes.io/projected/ade31a0f-9282-4bde-86a9-e17e7b41f05b-kube-api-access-2lrgd\") pod \"payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.442264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.442231    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7l8m\" (UniqueName: \"kubernetes.io/projected/6656cb26-447c-45e6-96c7-ab21cd9d6926-kube-api-access-j7l8m\") pod \"payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.442556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.442534    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6656cb26-447c-45e6-96c7-ab21cd9d6926-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.544057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.544019    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6656cb26-447c-45e6-96c7-ab21cd9d6926-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.544258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.544141    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7l8m\" (UniqueName: \"kubernetes.io/projected/6656cb26-447c-45e6-96c7-ab21cd9d6926-kube-api-access-j7l8m\") pod \"payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.544871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.544850    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6656cb26-447c-45e6-96c7-ab21cd9d6926-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.562980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.562955    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7l8m\" (UniqueName: \"kubernetes.io/projected/6656cb26-447c-45e6-96c7-ab21cd9d6926-kube-api-access-j7l8m\") pod \"payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.645548 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.645516    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:21.745229 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.745185    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:21.952182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:21.952145    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"]
Oct 30 13:50:22.060789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.060755    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"]
Oct 30 13:50:22.061181 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:22.060811    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6656cb26_447c_45e6_96c7_ab21cd9d6926.slice/crio-8a95d50e51c3440446141d56c0cb14872cc2ca51694caa67f8d9e8c42e3a21ff WatchSource:0}: Error finding container 8a95d50e51c3440446141d56c0cb14872cc2ca51694caa67f8d9e8c42e3a21ff: Status 404 returned error can't find the container with id 8a95d50e51c3440446141d56c0cb14872cc2ca51694caa67f8d9e8c42e3a21ff
Oct 30 13:50:22.585111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.585069    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" event={"ID":"6656cb26-447c-45e6-96c7-ab21cd9d6926","Type":"ContainerStarted","Data":"ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6"}
Oct 30 13:50:22.585361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.585130    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" event={"ID":"6656cb26-447c-45e6-96c7-ab21cd9d6926","Type":"ContainerStarted","Data":"8a95d50e51c3440446141d56c0cb14872cc2ca51694caa67f8d9e8c42e3a21ff"}
Oct 30 13:50:22.585361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.585163    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:22.586726 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.586700    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz" event={"ID":"ade31a0f-9282-4bde-86a9-e17e7b41f05b","Type":"ContainerStarted","Data":"34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e"}
Oct 30 13:50:22.586848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.586739    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz" event={"ID":"ade31a0f-9282-4bde-86a9-e17e7b41f05b","Type":"ContainerStarted","Data":"63e1357cfa6d2649894e65ee64a7bc35ecab87a29094b43f967a834096afaec9"}
Oct 30 13:50:22.586848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.586821    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:22.625452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.625397    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" podStartSLOduration=1.62538016 podStartE2EDuration="1.62538016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:22.624957086 +0000 UTC m=+2832.885195322" watchObservedRunningTime="2026-10-30 13:50:22.62538016 +0000 UTC m=+2832.885618399"
Oct 30 13:50:22.666720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:22.666670    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz" podStartSLOduration=2.66665686 podStartE2EDuration="2.66665686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:22.664107829 +0000 UTC m=+2832.924346064" watchObservedRunningTime="2026-10-30 13:50:22.66665686 +0000 UTC m=+2832.926895127"
Oct 30 13:50:25.006644 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.006605    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"]
Oct 30 13:50:25.011468 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.011450    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.025582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.025546    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:50:25.027026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.026999    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"]
Oct 30 13:50:25.033582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.033560    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:50:25.033582 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.033563    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:50:25.037739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.037709    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:50:25.095247 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.095214    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.095555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.095351    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm79g\" (UniqueName: \"kubernetes.io/projected/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-kube-api-access-cm79g\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.095555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.095404    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.196621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.196587    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.196857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.196725    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm79g\" (UniqueName: \"kubernetes.io/projected/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-kube-api-access-cm79g\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.196857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.196770    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.197465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.197442    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.199206 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.199182    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.217614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.217590    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm79g\" (UniqueName: \"kubernetes.io/projected/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-kube-api-access-cm79g\") pod \"maas-api-revoke-sub-c68ff24a-h9zbl\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.329447 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.329356    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:25.641806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:25.641772    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"]
Oct 30 13:50:25.644456 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:25.644428    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf25dd2fc_a9aa_4f84_b32e_dfc7e0790e1e.slice/crio-a3e973eb03d3bf553d933a351f18a1ae1ab7f2876f4b2621b9c9aca73bfad447 WatchSource:0}: Error finding container a3e973eb03d3bf553d933a351f18a1ae1ab7f2876f4b2621b9c9aca73bfad447: Status 404 returned error can't find the container with id a3e973eb03d3bf553d933a351f18a1ae1ab7f2876f4b2621b9c9aca73bfad447
Oct 30 13:50:26.620075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:26.620036    4291 generic.go:356] "Generic (PLEG): container finished" podID="f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" containerID="6a1feba40acf0d765ed16e653805f7d4f72bd27831d2b9d29c3bc4ee250c0a2b" exitCode=0
Oct 30 13:50:26.620571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:26.620112    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl" event={"ID":"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e","Type":"ContainerDied","Data":"6a1feba40acf0d765ed16e653805f7d4f72bd27831d2b9d29c3bc4ee250c0a2b"}
Oct 30 13:50:26.620571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:26.620181    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl" event={"ID":"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e","Type":"ContainerStarted","Data":"a3e973eb03d3bf553d933a351f18a1ae1ab7f2876f4b2621b9c9aca73bfad447"}
Oct 30 13:50:27.797732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.797707    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:27.841816 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.841783    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-kube-api-access-cm79g\" (UniqueName: \"kubernetes.io/projected/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-kube-api-access-cm79g\") pod \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") "
Oct 30 13:50:27.842006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.841863    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-xks-ca\") pod \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") "
Oct 30 13:50:27.842006 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.841913    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-openshift-ca\") pod \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\" (UID: \"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e\") "
Oct 30 13:50:27.842292 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.842265    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-openshift-ca" pod "f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" (UID: "f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:27.842391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.842377    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:27.844029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.844007    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-xks-ca" pod "f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" (UID: "f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:27.844287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.844258    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-kube-api-access-cm79g" pod "f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" (UID: "f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e"). InnerVolumeSpecName "kube-api-access-cm79g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:27.944555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.944512    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm79g\" (UniqueName: \"kubernetes.io/projected/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-kube-api-access-cm79g\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:27.944555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:27.944550    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:28.558793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:28.558763    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:50:28.558974 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:28.558837    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:50:28.647479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:28.647446    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"
Oct 30 13:50:28.647479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:28.647466    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl" event={"ID":"f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e","Type":"ContainerDied","Data":"a3e973eb03d3bf553d933a351f18a1ae1ab7f2876f4b2621b9c9aca73bfad447"}
Oct 30 13:50:28.647739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:28.647529    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3e973eb03d3bf553d933a351f18a1ae1ab7f2876f4b2621b9c9aca73bfad447"
Oct 30 13:50:33.596550 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:33.596520    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 13:50:33.597004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:33.596589    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 13:50:37.451196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.451159    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"]
Oct 30 13:50:37.451915 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.451896    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" containerName="revoke-keys"
Oct 30 13:50:37.451976 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.451916    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" containerName="revoke-keys"
Oct 30 13:50:37.452066 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.452056    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" containerName="revoke-keys"
Oct 30 13:50:37.456555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.456528    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.473919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.473884    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"]
Oct 30 13:50:37.480731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.480691    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-plugins-e2e-extmod-a-74589c\""
Oct 30 13:50:37.480731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.480700    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"openshift-service-ca.crt\""
Oct 30 13:50:37.480731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.480700    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"routing-overlay\""
Oct 30 13:50:37.481027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.480799    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"kube-root-ca.crt\""
Oct 30 13:50:37.484696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.484661    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"agc-cred-f06454846ef2\""
Oct 30 13:50:37.484875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.484747    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-74589c\"/\"payload-processing-external-model-e2e--1c277df7-dockercfg-59xr6\""
Oct 30 13:50:37.598373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.598326    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.598569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.598496    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.598615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.598599    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw8tx\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-kube-api-access-xw8tx\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.598730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.598669    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.685658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.684715    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"]
Oct 30 13:50:37.764946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.764861    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:37.768729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.768694    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.769587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.769561    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.775914 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.770576    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.776303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.776280    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw8tx\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-kube-api-access-xw8tx\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.805812 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.805770    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.817897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.785232    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:50:37.818134 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.785330    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:50:37.818222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.785950    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.830664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.785966    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:50:37.830938 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.796295    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"]
Oct 30 13:50:37.831089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.785062    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:50:37.831186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.806767    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.885921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.866357    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw8tx\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-kube-api-access-xw8tx\") pod \"payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") " pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:37.919471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.919432    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5t4z\" (UniqueName: \"kubernetes.io/projected/5f5a33f6-467a-4519-b629-34bba8a7c261-kube-api-access-r5t4z\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:37.919777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.919748    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:37.919925 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:37.919804    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.020959 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.020858    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5t4z\" (UniqueName: \"kubernetes.io/projected/5f5a33f6-467a-4519-b629-34bba8a7c261-kube-api-access-r5t4z\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.021141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.021054    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.021222 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.021187    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.021807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.021782    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.023617 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.023593    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.040897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.040854    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5t4z\" (UniqueName: \"kubernetes.io/projected/5f5a33f6-467a-4519-b629-34bba8a7c261-kube-api-access-r5t4z\") pod \"maas-api-revoke-sub-3250ea76-hjbmq\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.074112 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.074073    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:38.215667 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.214924    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:38.372461 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:50:38.372414    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:50:38.583498 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:38.583467    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f632bc5_e7fa_4c69_bf32_6e8508264c10.slice/crio-1bd06d91552e61815cd6d64b65bb73c6eeed11d4823c746e2c9bef26cba423f0 WatchSource:0}: Error finding container 1bd06d91552e61815cd6d64b65bb73c6eeed11d4823c746e2c9bef26cba423f0: Status 404 returned error can't find the container with id 1bd06d91552e61815cd6d64b65bb73c6eeed11d4823c746e2c9bef26cba423f0
Oct 30 13:50:38.583884 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.583702    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"]
Oct 30 13:50:38.635802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.635775    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"]
Oct 30 13:50:38.643273 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:38.643246    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f5a33f6_467a_4519_b629_34bba8a7c261.slice/crio-0cb32532c3cf4061b171920077efb5fec041151a995c529923f982a0a8fc73e7 WatchSource:0}: Error finding container 0cb32532c3cf4061b171920077efb5fec041151a995c529923f982a0a8fc73e7: Status 404 returned error can't find the container with id 0cb32532c3cf4061b171920077efb5fec041151a995c529923f982a0a8fc73e7
Oct 30 13:50:38.776919 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.776885    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq" event={"ID":"5f5a33f6-467a-4519-b629-34bba8a7c261","Type":"ContainerStarted","Data":"0cb32532c3cf4061b171920077efb5fec041151a995c529923f982a0a8fc73e7"}
Oct 30 13:50:38.778537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.778506    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm" event={"ID":"2f632bc5-e7fa-4c69-bf32-6e8508264c10","Type":"ContainerStarted","Data":"db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d"}
Oct 30 13:50:38.778708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.778551    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm" event={"ID":"2f632bc5-e7fa-4c69-bf32-6e8508264c10","Type":"ContainerStarted","Data":"1bd06d91552e61815cd6d64b65bb73c6eeed11d4823c746e2c9bef26cba423f0"}
Oct 30 13:50:38.778708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.778659    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:38.817739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:38.817670    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm" podStartSLOduration=1.817619538 podStartE2EDuration="1.817619538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:38.816603705 +0000 UTC m=+2849.076841943" watchObservedRunningTime="2026-10-30 13:50:38.817619538 +0000 UTC m=+2849.077857772"
Oct 30 13:50:39.786913 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:39.786879    4291 generic.go:356] "Generic (PLEG): container finished" podID="5f5a33f6-467a-4519-b629-34bba8a7c261" containerID="525ef135775adf8238a9b4f1ad4f6919d0e884672f2ae648f07a999029de0cff" exitCode=0
Oct 30 13:50:39.787390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:39.786971    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq" event={"ID":"5f5a33f6-467a-4519-b629-34bba8a7c261","Type":"ContainerDied","Data":"525ef135775adf8238a9b4f1ad4f6919d0e884672f2ae648f07a999029de0cff"}
Oct 30 13:50:40.965733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:40.965705    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:41.006603 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.006475    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-openshift-ca\") pod \"5f5a33f6-467a-4519-b629-34bba8a7c261\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") "
Oct 30 13:50:41.021463 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.021417    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f5a33f6-467a-4519-b629-34bba8a7c261-kube-api-access-r5t4z\" (UniqueName: \"kubernetes.io/projected/5f5a33f6-467a-4519-b629-34bba8a7c261-kube-api-access-r5t4z\") pod \"5f5a33f6-467a-4519-b629-34bba8a7c261\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") "
Oct 30 13:50:41.022096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.022072    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-xks-ca\") pod \"5f5a33f6-467a-4519-b629-34bba8a7c261\" (UID: \"5f5a33f6-467a-4519-b629-34bba8a7c261\") "
Oct 30 13:50:41.024906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.011086    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-openshift-ca" pod "5f5a33f6-467a-4519-b629-34bba8a7c261" (UID: "5f5a33f6-467a-4519-b629-34bba8a7c261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:41.046690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.046621    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f5a33f6-467a-4519-b629-34bba8a7c261-kube-api-access-r5t4z" pod "5f5a33f6-467a-4519-b629-34bba8a7c261" (UID: "5f5a33f6-467a-4519-b629-34bba8a7c261"). InnerVolumeSpecName "kube-api-access-r5t4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:41.047018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.046981    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-xks-ca" pod "5f5a33f6-467a-4519-b629-34bba8a7c261" (UID: "5f5a33f6-467a-4519-b629-34bba8a7c261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:41.048628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.048607    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:41.049477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.049459    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5t4z\" (UniqueName: \"kubernetes.io/projected/5f5a33f6-467a-4519-b629-34bba8a7c261-kube-api-access-r5t4z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:41.049618 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.049604    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f5a33f6-467a-4519-b629-34bba8a7c261-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:41.803933 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.803901    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"
Oct 30 13:50:41.804118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.803901    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq" event={"ID":"5f5a33f6-467a-4519-b629-34bba8a7c261","Type":"ContainerDied","Data":"0cb32532c3cf4061b171920077efb5fec041151a995c529923f982a0a8fc73e7"}
Oct 30 13:50:41.804118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:41.804027    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cb32532c3cf4061b171920077efb5fec041151a995c529923f982a0a8fc73e7"
Oct 30 13:50:48.807907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.807865    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"]
Oct 30 13:50:48.808686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.808668    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5f5a33f6-467a-4519-b629-34bba8a7c261" containerName="revoke-keys"
Oct 30 13:50:48.808686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.808686    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f5a33f6-467a-4519-b629-34bba8a7c261" containerName="revoke-keys"
Oct 30 13:50:48.808854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.808844    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f5a33f6-467a-4519-b629-34bba8a7c261" containerName="revoke-keys"
Oct 30 13:50:48.812920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.812903    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:48.827030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.826998    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:50:48.827141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.827116    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:50:48.830158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.830131    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"]
Oct 30 13:50:48.835576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.835554    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:50:48.835729 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.835655    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:50:48.911393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.911360    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:48.911557 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.911539    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:48.911596 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:48.911585    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwmjc\" (UniqueName: \"kubernetes.io/projected/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-kube-api-access-xwmjc\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.012762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.012730    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.012945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.012823    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.012945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.012844    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwmjc\" (UniqueName: \"kubernetes.io/projected/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-kube-api-access-xwmjc\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.013568 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.013544    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.015147 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.015124    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.032734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.032704    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwmjc\" (UniqueName: \"kubernetes.io/projected/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-kube-api-access-xwmjc\") pod \"maas-api-revoke-sub-42e7724e-5jwd7\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.131527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.131488    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:49.443278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.443248    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"]
Oct 30 13:50:49.443826 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:50:49.443801    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20e0ea8d_bc7b_4c46_8f23_b48697dc9ade.slice/crio-57246d29e19ca3176e7f4f9eb82f6d7ca149d1eebcd74189bb126e5a5f50c187 WatchSource:0}: Error finding container 57246d29e19ca3176e7f4f9eb82f6d7ca149d1eebcd74189bb126e5a5f50c187: Status 404 returned error can't find the container with id 57246d29e19ca3176e7f4f9eb82f6d7ca149d1eebcd74189bb126e5a5f50c187
Oct 30 13:50:49.790612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.790581    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:50:49.884720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.884680    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7" event={"ID":"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade","Type":"ContainerStarted","Data":"82893d111fe07da040b9b81fc87fc2edbf601a4fe78bf8041d8a34e127585a7a"}
Oct 30 13:50:49.885327 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.885301    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7" event={"ID":"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade","Type":"ContainerStarted","Data":"57246d29e19ca3176e7f4f9eb82f6d7ca149d1eebcd74189bb126e5a5f50c187"}
Oct 30 13:50:49.937025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:49.936954    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7" podStartSLOduration=1.936932882 podStartE2EDuration="1.936932882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:50:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:50:49.930578158 +0000 UTC m=+2860.190816407" watchObservedRunningTime="2026-10-30 13:50:49.936932882 +0000 UTC m=+2860.197171117"
Oct 30 13:50:50.673096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:50.673053    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"]
Oct 30 13:50:50.892881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:50.892848    4291 generic.go:356] "Generic (PLEG): container finished" podID="20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" containerID="82893d111fe07da040b9b81fc87fc2edbf601a4fe78bf8041d8a34e127585a7a" exitCode=0
Oct 30 13:50:50.893294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:50.892926    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7" event={"ID":"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade","Type":"ContainerDied","Data":"82893d111fe07da040b9b81fc87fc2edbf601a4fe78bf8041d8a34e127585a7a"}
Oct 30 13:50:52.067102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.067075    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:52.182307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.182209    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-xks-ca\") pod \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") "
Oct 30 13:50:52.182494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.182341    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-openshift-ca\") pod \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") "
Oct 30 13:50:52.182494 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.182482    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-kube-api-access-xwmjc\" (UniqueName: \"kubernetes.io/projected/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-kube-api-access-xwmjc\") pod \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\" (UID: \"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade\") "
Oct 30 13:50:52.182769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.182734    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-openshift-ca" pod "20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" (UID: "20e0ea8d-bc7b-4c46-8f23-b48697dc9ade"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:52.183057 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.183022    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:52.184601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.184578    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-xks-ca" pod "20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" (UID: "20e0ea8d-bc7b-4c46-8f23-b48697dc9ade"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:50:52.184695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.184618    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-kube-api-access-xwmjc" pod "20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" (UID: "20e0ea8d-bc7b-4c46-8f23-b48697dc9ade"). InnerVolumeSpecName "kube-api-access-xwmjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:52.284063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.284026    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwmjc\" (UniqueName: \"kubernetes.io/projected/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-kube-api-access-xwmjc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:52.284063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.284060    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:52.915118 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.915076    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7" event={"ID":"20e0ea8d-bc7b-4c46-8f23-b48697dc9ade","Type":"ContainerDied","Data":"57246d29e19ca3176e7f4f9eb82f6d7ca149d1eebcd74189bb126e5a5f50c187"}
Oct 30 13:50:52.915373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.915145    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57246d29e19ca3176e7f4f9eb82f6d7ca149d1eebcd74189bb126e5a5f50c187"
Oct 30 13:50:52.915373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:52.915221    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"
Oct 30 13:50:54.089594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:54.089559    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"]
Oct 30 13:50:54.100598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:54.100565    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d46f453e-rkp7r"]
Oct 30 13:50:55.673805 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:55.673736    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9" podUID="d6f5c474-4e88-4eb2-ba77-2d9e7658c150" containerName="payload-processing" containerID="cri-o://12fd99b10d266618d954cfa0cee887954d567b0d79b92fcd4285de7b38f4e4ef" gracePeriod=25
Oct 30 13:50:55.883071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:55.883039    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a37256d-b608-425e-9b04-0b0db90480b7" path="/var/lib/kubelet/pods/5a37256d-b608-425e-9b04-0b0db90480b7/volumes"
Oct 30 13:50:55.940533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:55.940504    4291 generic.go:356] "Generic (PLEG): container finished" podID="d6f5c474-4e88-4eb2-ba77-2d9e7658c150" containerID="12fd99b10d266618d954cfa0cee887954d567b0d79b92fcd4285de7b38f4e4ef" exitCode=0
Oct 30 13:50:55.940708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:55.940572    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9" event={"ID":"d6f5c474-4e88-4eb2-ba77-2d9e7658c150","Type":"ContainerDied","Data":"12fd99b10d266618d954cfa0cee887954d567b0d79b92fcd4285de7b38f4e4ef"}
Oct 30 13:50:55.940708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:55.940670    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9" event={"ID":"d6f5c474-4e88-4eb2-ba77-2d9e7658c150","Type":"ContainerDied","Data":"1f4fe68e2bc137e6da7e5fb1712d6362a564c28114c4fb291018ba6b221039e2"}
Oct 30 13:50:55.940708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:55.940699    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f4fe68e2bc137e6da7e5fb1712d6362a564c28114c4fb291018ba6b221039e2"
Oct 30 13:50:55.957072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:55.957045    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:50:56.036155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.036110    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-provider-credentials\") pod \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") "
Oct 30 13:50:56.036370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.036317    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-routing-overlay-volume\") pod \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") "
Oct 30 13:50:56.036370 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.036366    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-kube-api-access-xltpn\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-kube-api-access-xltpn\") pod \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") "
Oct 30 13:50:56.036556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.036536    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-plugins-config-volume\") pod \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\" (UID: \"d6f5c474-4e88-4eb2-ba77-2d9e7658c150\") "
Oct 30 13:50:56.036831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.036797    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-routing-overlay-volume" pod "d6f5c474-4e88-4eb2-ba77-2d9e7658c150" (UID: "d6f5c474-4e88-4eb2-ba77-2d9e7658c150"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:56.037223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.037197    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-plugins-config-volume" pod "d6f5c474-4e88-4eb2-ba77-2d9e7658c150" (UID: "d6f5c474-4e88-4eb2-ba77-2d9e7658c150"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:50:56.037417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.037393    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:56.037492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.037424    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:56.038809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.038778    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-provider-credentials" pod "d6f5c474-4e88-4eb2-ba77-2d9e7658c150" (UID: "d6f5c474-4e88-4eb2-ba77-2d9e7658c150"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:56.039137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.039119    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-kube-api-access-xltpn" pod "d6f5c474-4e88-4eb2-ba77-2d9e7658c150" (UID: "d6f5c474-4e88-4eb2-ba77-2d9e7658c150"). InnerVolumeSpecName "kube-api-access-xltpn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:50:56.141075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.141028    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:56.141075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.141075    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xltpn\" (UniqueName: \"kubernetes.io/projected/d6f5c474-4e88-4eb2-ba77-2d9e7658c150-kube-api-access-xltpn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:50:56.947778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.947747    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"
Oct 30 13:50:56.999561 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:56.999526    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"]
Oct 30 13:50:57.011265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:57.011230    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-98f1cc/payload-processing-external-model-e2e-extmod-b-98f1cc-74c8pdps9"]
Oct 30 13:50:57.881583 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:50:57.881552    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6f5c474-4e88-4eb2-ba77-2d9e7658c150" path="/var/lib/kubelet/pods/d6f5c474-4e88-4eb2-ba77-2d9e7658c150/volumes"
Oct 30 13:51:00.457320 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:00.457277    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"]
Oct 30 13:51:05.457901 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.457688    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm" podUID="2f632bc5-e7fa-4c69-bf32-6e8508264c10" containerName="payload-processing" containerID="cri-o://db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d" gracePeriod=25
Oct 30 13:51:05.757683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.757653    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:51:05.819373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.819319    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-provider-credentials\") pod \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") "
Oct 30 13:51:05.819810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.819743    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-routing-overlay-volume\") pod \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") "
Oct 30 13:51:05.820124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.820062    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-kube-api-access-xw8tx\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-kube-api-access-xw8tx\") pod \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") "
Oct 30 13:51:05.820518 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.820470    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-plugins-config-volume\") pod \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\" (UID: \"2f632bc5-e7fa-4c69-bf32-6e8508264c10\") "
Oct 30 13:51:05.821803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.821766    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-plugins-config-volume" pod "2f632bc5-e7fa-4c69-bf32-6e8508264c10" (UID: "2f632bc5-e7fa-4c69-bf32-6e8508264c10"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:05.822553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.822522    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-routing-overlay-volume" pod "2f632bc5-e7fa-4c69-bf32-6e8508264c10" (UID: "2f632bc5-e7fa-4c69-bf32-6e8508264c10"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:51:05.856706 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.856608    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-provider-credentials" pod "2f632bc5-e7fa-4c69-bf32-6e8508264c10" (UID: "2f632bc5-e7fa-4c69-bf32-6e8508264c10"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:05.863967 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.863926    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-kube-api-access-xw8tx" pod "2f632bc5-e7fa-4c69-bf32-6e8508264c10" (UID: "2f632bc5-e7fa-4c69-bf32-6e8508264c10"). InnerVolumeSpecName "kube-api-access-xw8tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:51:05.926303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.925545    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw8tx\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-kube-api-access-xw8tx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:51:05.926303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.925577    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:51:05.926303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.925593    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/2f632bc5-e7fa-4c69-bf32-6e8508264c10-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:51:05.926303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:05.925607    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/2f632bc5-e7fa-4c69-bf32-6e8508264c10-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:51:06.025209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.025120    4291 generic.go:356] "Generic (PLEG): container finished" podID="2f632bc5-e7fa-4c69-bf32-6e8508264c10" containerID="db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d" exitCode=0
Oct 30 13:51:06.025209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.025167    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"
Oct 30 13:51:06.025408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.025198    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm" event={"ID":"2f632bc5-e7fa-4c69-bf32-6e8508264c10","Type":"ContainerDied","Data":"db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d"}
Oct 30 13:51:06.025408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.025262    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm" event={"ID":"2f632bc5-e7fa-4c69-bf32-6e8508264c10","Type":"ContainerDied","Data":"1bd06d91552e61815cd6d64b65bb73c6eeed11d4823c746e2c9bef26cba423f0"}
Oct 30 13:51:06.025408 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.025291    4291 scope.go:122] "RemoveContainer" containerID="db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d"
Oct 30 13:51:06.035843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.035817    4291 scope.go:122] "RemoveContainer" containerID="db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d"
Oct 30 13:51:06.036123 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:51:06.036101    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d\": container with ID starting with db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d not found: ID does not exist" containerID="db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d"
Oct 30 13:51:06.036197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.036131    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d"} err="failed to get container status \"db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d\": rpc error: code = NotFound desc = could not find container \"db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d\": container with ID starting with db93ee39dc813e487686b3e17c4f910567ea21f461f07e9f9203886edfe8468d not found: ID does not exist"
Oct 30 13:51:06.079314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.079276    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"]
Oct 30 13:51:06.083600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.083561    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-74589c/payload-processing-external-model-e2e-extmod-a-74589c-76b7nlrdm"]
Oct 30 13:51:06.585027 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:06.584990    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:51:07.880928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:07.880892    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f632bc5-e7fa-4c69-bf32-6e8508264c10" path="/var/lib/kubelet/pods/2f632bc5-e7fa-4c69-bf32-6e8508264c10/volumes"
Oct 30 13:51:08.224897 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:51:08.224796    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:51:13.236086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:13.236053    4291 scope.go:122] "RemoveContainer" containerID="0f3bd4e9069d3ca93d8e529ed209b70d7bf5640eea3c2ff6454bbaf610efae23"
Oct 30 13:51:13.246892 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:13.246873    4291 scope.go:122] "RemoveContainer" containerID="f7b9d27bb33453a507e6bab9c0680a7d1a1b0b802c7806907cfa5eec405b2510"
Oct 30 13:51:14.553522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.553486    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"]
Oct 30 13:51:14.554208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554184    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2f632bc5-e7fa-4c69-bf32-6e8508264c10" containerName="payload-processing"
Oct 30 13:51:14.554208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554204    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f632bc5-e7fa-4c69-bf32-6e8508264c10" containerName="payload-processing"
Oct 30 13:51:14.554416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554230    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d6f5c474-4e88-4eb2-ba77-2d9e7658c150" containerName="payload-processing"
Oct 30 13:51:14.554416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554235    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6f5c474-4e88-4eb2-ba77-2d9e7658c150" containerName="payload-processing"
Oct 30 13:51:14.554416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554249    4291 cpu_manager.go:414] "containerMap: removing container" podUID="20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" containerName="revoke-keys"
Oct 30 13:51:14.554416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554254    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" containerName="revoke-keys"
Oct 30 13:51:14.554416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554371    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f632bc5-e7fa-4c69-bf32-6e8508264c10" containerName="payload-processing"
Oct 30 13:51:14.554416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554384    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6f5c474-4e88-4eb2-ba77-2d9e7658c150" containerName="payload-processing"
Oct 30 13:51:14.554416 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.554391    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" containerName="revoke-keys"
Oct 30 13:51:14.559831 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.559802    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:14.575202 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.575174    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"]
Oct 30 13:51:14.580840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.580819    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-926ff0\""
Oct 30 13:51:14.632981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.632950    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"]
Oct 30 13:51:14.637512 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.637490    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:14.652195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.652168    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-926ff0-dockercfg-d4cgd\""
Oct 30 13:51:14.656710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.656682    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"]
Oct 30 13:51:14.762092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.740184    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b822f93-3cd3-411d-a127-fe2df056c84b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:14.762422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.762385    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglbk\" (UniqueName: \"kubernetes.io/projected/4b822f93-3cd3-411d-a127-fe2df056c84b-kube-api-access-zglbk\") pod \"payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:14.893394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.893349    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b822f93-3cd3-411d-a127-fe2df056c84b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:14.893605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.893403    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglbk\" (UniqueName: \"kubernetes.io/projected/4b822f93-3cd3-411d-a127-fe2df056c84b-kube-api-access-zglbk\") pod \"payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:14.893605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.893578    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:14.895792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.895765    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkkjs\" (UniqueName: \"kubernetes.io/projected/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-kube-api-access-rkkjs\") pod \"payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:14.897514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.896747    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b822f93-3cd3-411d-a127-fe2df056c84b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:14.932415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.929430    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglbk\" (UniqueName: \"kubernetes.io/projected/4b822f93-3cd3-411d-a127-fe2df056c84b-kube-api-access-zglbk\") pod \"payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:14.998412 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.998376    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkkjs\" (UniqueName: \"kubernetes.io/projected/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-kube-api-access-rkkjs\") pod \"payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:14.998670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.998653    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:14.999319 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:14.999299    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:15.018377 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:15.018339    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkkjs\" (UniqueName: \"kubernetes.io/projected/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-kube-api-access-rkkjs\") pod \"payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:15.176803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:15.176710    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:15.256839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:15.252097    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:15.492328 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:51:15.492300    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b822f93_3cd3_411d_a127_fe2df056c84b.slice/crio-68ac440e5d0b4f0ba2b0fd3c4659067a90551bbf9444b3341d4e356b0a98b0af WatchSource:0}: Error finding container 68ac440e5d0b4f0ba2b0fd3c4659067a90551bbf9444b3341d4e356b0a98b0af: Status 404 returned error can't find the container with id 68ac440e5d0b4f0ba2b0fd3c4659067a90551bbf9444b3341d4e356b0a98b0af
Oct 30 13:51:15.492439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:15.492344    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"]
Oct 30 13:51:15.582675 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:15.582650    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"]
Oct 30 13:51:15.584464 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:51:15.584441    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e806a3c_b759_4e87_89b7_9bb0a7c911c9.slice/crio-c1e56bec41d4fa51ebdc43a3b37e04ea1c9efa56e2755129f660d9306febda17 WatchSource:0}: Error finding container c1e56bec41d4fa51ebdc43a3b37e04ea1c9efa56e2755129f660d9306febda17: Status 404 returned error can't find the container with id c1e56bec41d4fa51ebdc43a3b37e04ea1c9efa56e2755129f660d9306febda17
Oct 30 13:51:16.128502 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.128464    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" event={"ID":"9e806a3c-b759-4e87-89b7-9bb0a7c911c9","Type":"ContainerStarted","Data":"abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8"}
Oct 30 13:51:16.128764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.128526    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" event={"ID":"9e806a3c-b759-4e87-89b7-9bb0a7c911c9","Type":"ContainerStarted","Data":"c1e56bec41d4fa51ebdc43a3b37e04ea1c9efa56e2755129f660d9306febda17"}
Oct 30 13:51:16.128764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.128553    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:16.130314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.130280    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g" event={"ID":"4b822f93-3cd3-411d-a127-fe2df056c84b","Type":"ContainerStarted","Data":"d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0"}
Oct 30 13:51:16.130431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.130332    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g" event={"ID":"4b822f93-3cd3-411d-a127-fe2df056c84b","Type":"ContainerStarted","Data":"68ac440e5d0b4f0ba2b0fd3c4659067a90551bbf9444b3341d4e356b0a98b0af"}
Oct 30 13:51:16.130431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.130366    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:16.168404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.168342    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" podStartSLOduration=2.168329427 podStartE2EDuration="2.168329427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:16.165433637 +0000 UTC m=+2886.425671875" watchObservedRunningTime="2026-10-30 13:51:16.168329427 +0000 UTC m=+2886.428567661"
Oct 30 13:51:16.205593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:16.205536    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g" podStartSLOduration=2.205523483 podStartE2EDuration="2.205523483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:51:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:51:16.201709204 +0000 UTC m=+2886.461947440" watchObservedRunningTime="2026-10-30 13:51:16.205523483 +0000 UTC m=+2886.465761716"
Oct 30 13:51:27.140902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:27.140867    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 13:51:27.143361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:27.140942    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 13:51:38.212847 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:51:38.212800    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:51:43.059758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:43.059699    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"]
Oct 30 13:51:43.071392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:43.071355    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"]
Oct 30 13:51:43.075521 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:43.075482    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d9f6a4f0-t7nb4"]
Oct 30 13:51:43.079311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:43.079284    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bdb85861-gvlqc"]
Oct 30 13:51:43.885549 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:43.885516    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f5d8c57-8a0e-49ff-b029-51189c5dc404" path="/var/lib/kubelet/pods/9f5d8c57-8a0e-49ff-b029-51189c5dc404/volumes"
Oct 30 13:51:43.886069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:43.886054    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca7f724-6a93-4a97-a9cf-994fead4d010" path="/var/lib/kubelet/pods/bca7f724-6a93-4a97-a9cf-994fead4d010/volumes"
Oct 30 13:51:54.052137 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:54.052041    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"]
Oct 30 13:51:54.063043 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:54.063005    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b820a66-jzhg5"]
Oct 30 13:51:54.067029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:54.066999    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"]
Oct 30 13:51:54.070683 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:54.070659    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b37c58c-sqm49"]
Oct 30 13:51:55.883614 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:55.883575    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb" path="/var/lib/kubelet/pods/496f9c7e-6ef7-4b44-b8d5-a7f641af1ffb/volumes"
Oct 30 13:51:55.884143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:51:55.884126    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0b3038d-1d24-458c-ad51-6ac9dfe17d85" path="/var/lib/kubelet/pods/b0b3038d-1d24-458c-ad51-6ac9dfe17d85/volumes"
Oct 30 13:52:02.949759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:02.949702    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:52:08.213509 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:52:08.213469    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:52:09.687698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.687656    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"]
Oct 30 13:52:09.694841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.694809    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.717523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.717491    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"]
Oct 30 13:52:09.724589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.724555    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5494c9\"/\"kube-root-ca.crt\""
Oct 30 13:52:09.724803 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.724719    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5494c9\"/\"test-model-5494c9-kserve-self-signed-certs\""
Oct 30 13:52:09.724906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.724833    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5494c9\"/\"default-dockercfg-b9rl2\""
Oct 30 13:52:09.724906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.724873    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-5494c9\"/\"openshift-service-ca.crt\""
Oct 30 13:52:09.773304 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.773270    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tmp-dir\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.773474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.773316    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-model-cache\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.773474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.773429    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tls-certs\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.773559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.773515    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdr7f\" (UniqueName: \"kubernetes.io/projected/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-kube-api-access-xdr7f\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.773608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.773593    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-dshm\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.773692 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.773677    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-home\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.874706    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-model-cache\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.874787    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tls-certs\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.874892    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdr7f\" (UniqueName: \"kubernetes.io/projected/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-kube-api-access-xdr7f\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.875027    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-dshm\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.875077    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-home\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.875210    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tmp-dir\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.875256    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-model-cache\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.875549    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-home\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.876032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.875616    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tmp-dir\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.879599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.879514    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tls-certs\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.885153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.885122    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-dshm\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:09.900198 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:09.900171    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdr7f\" (UniqueName: \"kubernetes.io/projected/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-kube-api-access-xdr7f\") pod \"test-model-5494c9-kserve-7df459fbdc-m8hpv\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") " pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:10.026143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:10.026050    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-5494c9\"/\"default-dockercfg-b9rl2\""
Oct 30 13:52:10.031897 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:10.031577    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:10.447858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:10.447819    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"]
Oct 30 13:52:10.451804 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:52:10.451769    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7eaa41c_2d83_46d3_a360_904c3b46d0b9.slice/crio-f884dbb8aa95727496151d500dd6634775769f2e2edaf2a49c4aecd9e536f455 WatchSource:0}: Error finding container f884dbb8aa95727496151d500dd6634775769f2e2edaf2a49c4aecd9e536f455: Status 404 returned error can't find the container with id f884dbb8aa95727496151d500dd6634775769f2e2edaf2a49c4aecd9e536f455
Oct 30 13:52:10.683811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:10.683763    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv" event={"ID":"f7eaa41c-2d83-46d3-a360-904c3b46d0b9","Type":"ContainerStarted","Data":"9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898"}
Oct 30 13:52:10.683811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:10.683824    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv" event={"ID":"f7eaa41c-2d83-46d3-a360-904c3b46d0b9","Type":"ContainerStarted","Data":"f884dbb8aa95727496151d500dd6634775769f2e2edaf2a49c4aecd9e536f455"}
Oct 30 13:52:10.797058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:10.796951    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv" podStartSLOduration=1.796936107 podStartE2EDuration="1.796936107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:10.794467288 +0000 UTC m=+2941.054705522" watchObservedRunningTime="2026-10-30 13:52:10.796936107 +0000 UTC m=+2941.057174342"
Oct 30 13:52:11.033274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:11.033231    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:13.103508 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.103419    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"]
Oct 30 13:52:13.147659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.144925    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"]
Oct 30 13:52:13.147659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.145258    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.178546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.178505    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-09014b-openshift-default-dockercfg-9mn6d\""
Oct 30 13:52:13.295279 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.295241    4291 scope.go:122] "RemoveContainer" containerID="a146065da0f37cf2ae18f9590bb0f8d13c32958a255c0e772dfa304de13ff95e"
Oct 30 13:52:13.332368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332322    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-socket\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.332591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332411    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-credential-socket\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.332591 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332436    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-podinfo\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.332739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332582    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-token\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.332739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332674    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-certs\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.332739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332719    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a6bbcd-4039-492d-8c6b-079648353ef2-istiod-ca-cert\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.332904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332809    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-data\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.332904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332892    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-envoy\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.333008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.332986    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s2wjx\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-kube-api-access-s2wjx\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434282    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a6bbcd-4039-492d-8c6b-079648353ef2-istiod-ca-cert\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434348    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-data\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434415    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-envoy\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434494    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s2wjx\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-kube-api-access-s2wjx\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434654    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-socket\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434718    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-credential-socket\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434747    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-podinfo\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434844    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-data\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.434881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434851    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-token\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.435107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.434933    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-certs\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.435107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.435050    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-socket\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.435107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.435095    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a6bbcd-4039-492d-8c6b-079648353ef2-istiod-ca-cert\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.435254 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.435224    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-certs\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.435381 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.435363    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-credential-socket\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.437487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.437460    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-envoy\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.437616 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.437563    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-podinfo\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.457954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.457913    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s2wjx\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-kube-api-access-s2wjx\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.457954 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.457947    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-token\") pod \"e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") " pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.475684 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.475630    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:13.843185 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.843112    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"]
Oct 30 13:52:13.846310 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:52:13.846263    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a6bbcd_4039_492d_8c6b_079648353ef2.slice/crio-9a9b4c361da03639d8035673aad85c480daae9c1918cc63be1ab6b1c726e64b8 WatchSource:0}: Error finding container 9a9b4c361da03639d8035673aad85c480daae9c1918cc63be1ab6b1c726e64b8: Status 404 returned error can't find the container with id 9a9b4c361da03639d8035673aad85c480daae9c1918cc63be1ab6b1c726e64b8
Oct 30 13:52:13.849745 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.849709    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:13.849835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.849783    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:13.849835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:13.849814    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:52:14.721558 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:14.721508    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" event={"ID":"31a6bbcd-4039-492d-8c6b-079648353ef2","Type":"ContainerStarted","Data":"ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa"}
Oct 30 13:52:14.722042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:14.721579    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" event={"ID":"31a6bbcd-4039-492d-8c6b-079648353ef2","Type":"ContainerStarted","Data":"9a9b4c361da03639d8035673aad85c480daae9c1918cc63be1ab6b1c726e64b8"}
Oct 30 13:52:14.803380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:14.803322    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" podStartSLOduration=2.803304127 podStartE2EDuration="2.803304127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:14.79004184 +0000 UTC m=+2945.050280075" watchObservedRunningTime="2026-10-30 13:52:14.803304127 +0000 UTC m=+2945.063542360"
Oct 30 13:52:15.475920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:15.475876    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:15.482996 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:15.482957    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:15.728857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:15.728775    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:15.729861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:15.729839    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:52:19.411986 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.411950    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"]
Oct 30 13:52:19.416859 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.416837    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.438869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.438839    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"]
Oct 30 13:52:19.445158 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.445123    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-09014b\""
Oct 30 13:52:19.445442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.445380    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-09014b-dockercfg-55wgv\""
Oct 30 13:52:19.454492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.454459    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"]
Oct 30 13:52:19.460352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.460324    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:19.483155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.483122    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"]
Oct 30 13:52:19.558068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.555758    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.558068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.555937    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8rd8\" (UniqueName: \"kubernetes.io/projected/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-kube-api-access-p8rd8\") pod \"payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.558068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.556013    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:19.558068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.556109    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jcqh\" (UniqueName: \"kubernetes.io/projected/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-kube-api-access-6jcqh\") pod \"payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:19.657659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.657593    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jcqh\" (UniqueName: \"kubernetes.io/projected/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-kube-api-access-6jcqh\") pod \"payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:19.664704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.658584    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.665113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.665087    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8rd8\" (UniqueName: \"kubernetes.io/projected/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-kube-api-access-p8rd8\") pod \"payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.665761 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.665733    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:19.677656 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.666863    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:19.677879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.664834    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.729908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.721300    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8rd8\" (UniqueName: \"kubernetes.io/projected/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-kube-api-access-p8rd8\") pod \"payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.732334 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.732302    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jcqh\" (UniqueName: \"kubernetes.io/projected/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-kube-api-access-6jcqh\") pod \"payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") " pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:19.732506 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.732420    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:19.785234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:19.783935    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:20.032811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.032770    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:20.045946 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.045914    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:20.108514 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.108475    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"]
Oct 30 13:52:20.109187 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:52:20.109156    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d4333dd_8bbb_4304_abe9_f6a1fadbe36a.slice/crio-1761ce4eacdba8783247209a0f128239cc1529a02c8e008d999cf7e46dfe0701 WatchSource:0}: Error finding container 1761ce4eacdba8783247209a0f128239cc1529a02c8e008d999cf7e46dfe0701: Status 404 returned error can't find the container with id 1761ce4eacdba8783247209a0f128239cc1529a02c8e008d999cf7e46dfe0701
Oct 30 13:52:20.186036 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.185996    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"]
Oct 30 13:52:20.203328 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:52:20.203294    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d3b37cb_3117_4dce_9f23_b9eae6eea81f.slice/crio-56ef4c4680a3d30a1716cbe4d7b1f99f021f5c4fcee373bd665d3d8682cdda69 WatchSource:0}: Error finding container 56ef4c4680a3d30a1716cbe4d7b1f99f021f5c4fcee373bd665d3d8682cdda69: Status 404 returned error can't find the container with id 56ef4c4680a3d30a1716cbe4d7b1f99f021f5c4fcee373bd665d3d8682cdda69
Oct 30 13:52:20.806902 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.806833    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs" event={"ID":"8d3b37cb-3117-4dce-9f23-b9eae6eea81f","Type":"ContainerStarted","Data":"b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156"}
Oct 30 13:52:20.807375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.806942    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs" event={"ID":"8d3b37cb-3117-4dce-9f23-b9eae6eea81f","Type":"ContainerStarted","Data":"56ef4c4680a3d30a1716cbe4d7b1f99f021f5c4fcee373bd665d3d8682cdda69"}
Oct 30 13:52:20.807375 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.806973    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:20.808723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.808691    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" event={"ID":"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a","Type":"ContainerStarted","Data":"d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed"}
Oct 30 13:52:20.808887 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.808750    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" event={"ID":"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a","Type":"ContainerStarted","Data":"1761ce4eacdba8783247209a0f128239cc1529a02c8e008d999cf7e46dfe0701"}
Oct 30 13:52:20.820268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.820232    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:52:20.859774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.859719    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs" podStartSLOduration=1.8597025139999999 podStartE2EDuration="1.859702514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:20.856903328 +0000 UTC m=+2951.117141563" watchObservedRunningTime="2026-10-30 13:52:20.859702514 +0000 UTC m=+2951.119940778"
Oct 30 13:52:20.946125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:20.946063    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" podStartSLOduration=1.946042384 podStartE2EDuration="1.946042384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:52:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:52:20.937680969 +0000 UTC m=+2951.197919197" watchObservedRunningTime="2026-10-30 13:52:20.946042384 +0000 UTC m=+2951.206280619"
Oct 30 13:52:21.818526 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:21.818491    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:31.821759 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:31.821727    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 13:52:32.829977 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:32.829944    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 13:52:38.214713 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:52:38.214670    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:52:54.034160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:54.034112    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"]
Oct 30 13:52:54.051487 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:54.051442    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c375bafb-tn72c"]
Oct 30 13:52:55.038368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:55.038330    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"]
Oct 30 13:52:55.050048 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:55.050014    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8384f11-42wm6"]
Oct 30 13:52:55.881103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:55.881066    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1" path="/var/lib/kubelet/pods/02ae8c03-ffbe-42ff-aab2-e6e71c7a77a1/volumes"
Oct 30 13:52:55.881622 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:52:55.881606    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d9f2a73-0e1e-4ab7-a692-49a26d640686" path="/var/lib/kubelet/pods/1d9f2a73-0e1e-4ab7-a692-49a26d640686/volumes"
Oct 30 13:53:06.042380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:06.042342    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"]
Oct 30 13:53:06.053072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:06.053039    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-83166978-m2nj2"]
Oct 30 13:53:07.880857 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:07.880819    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90ec351a-c373-4b50-9632-1d7affadb8b1" path="/var/lib/kubelet/pods/90ec351a-c373-4b50-9632-1d7affadb8b1/volumes"
Oct 30 13:53:08.213396 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:53:08.213313    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:53:11.323785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.323740    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"]
Oct 30 13:53:11.330645 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.330607    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.358091 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.358057    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-319179-openshift-default-dockercfg-zfdvf\""
Oct 30 13:53:11.380156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.380122    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"]
Oct 30 13:53:11.396342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396087    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-certs\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396211    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-envoy\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396342 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396267    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-data\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396479    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-credential-socket\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396566    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a0ac3-b411-46d3-894c-0855bbeab4a6-istiod-ca-cert\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396621    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxdcr\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-kube-api-access-nxdcr\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396740    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-token\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396772    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-socket\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.396908 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.396810    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-podinfo\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.497819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.497778    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-credential-socket\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.497861    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a0ac3-b411-46d3-894c-0855bbeab4a6-istiod-ca-cert\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.497915    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxdcr\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-kube-api-access-nxdcr\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.497981    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-token\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.498009    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-socket\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.498047    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-podinfo\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.498097    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-certs\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.498162    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-envoy\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498289 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.498211    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-data\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.498723    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-data\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.498858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.498774    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-credential-socket\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.499265 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.499231    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-socket\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.499569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.499547    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-certs\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.499807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.499781    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a0ac3-b411-46d3-894c-0855bbeab4a6-istiod-ca-cert\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.502274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.502253    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-envoy\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.502428 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.502277    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-podinfo\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.519269 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.519246    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-token\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.519414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.519372    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxdcr\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-kube-api-access-nxdcr\") pod \"e2e-inf-b-319179-openshift-default-568ff76ccc-9545v\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") " pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:11.656661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:11.655994    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:12.154233 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.154203    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"]
Oct 30 13:53:12.155622 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:53:12.155595    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769a0ac3_b411_46d3_894c_0855bbeab4a6.slice/crio-6b1f7ec93bbbf10e275f48d701d177e371b6663d06ea3235bad62919637b7354 WatchSource:0}: Error finding container 6b1f7ec93bbbf10e275f48d701d177e371b6663d06ea3235bad62919637b7354: Status 404 returned error can't find the container with id 6b1f7ec93bbbf10e275f48d701d177e371b6663d06ea3235bad62919637b7354
Oct 30 13:53:12.159378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.159344    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:53:12.159467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.159412    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:53:12.159467 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.159442    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:53:12.426316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.426151    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" event={"ID":"769a0ac3-b411-46d3-894c-0855bbeab4a6","Type":"ContainerStarted","Data":"264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d"}
Oct 30 13:53:12.426316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.426226    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" event={"ID":"769a0ac3-b411-46d3-894c-0855bbeab4a6","Type":"ContainerStarted","Data":"6b1f7ec93bbbf10e275f48d701d177e371b6663d06ea3235bad62919637b7354"}
Oct 30 13:53:12.468107 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.468061    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:53:12.472217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.472138    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podStartSLOduration=1.472119618 podStartE2EDuration="1.472119618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:12.471079962 +0000 UTC m=+3002.731318197" watchObservedRunningTime="2026-10-30 13:53:12.472119618 +0000 UTC m=+3002.732357853"
Oct 30 13:53:12.660779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:12.660744    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:13.327678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:13.327630    4291 scope.go:122] "RemoveContainer" containerID="66523595d5d748f0507d8c614bfc50eed4232458943dcc6780a672dfb88f9d5f"
Oct 30 13:53:13.338766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:13.338740    4291 scope.go:122] "RemoveContainer" containerID="93850b2f4e0d79c0b0df545e8d76a43568abbd20e908b96f10843ada46443a99"
Oct 30 13:53:13.349008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:13.348986    4291 scope.go:122] "RemoveContainer" containerID="f5459c778debe5d4b0120e14755f9005635dbe969895b15ea3feb40f91b6f7aa"
Oct 30 13:53:13.359115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:13.359092    4291 scope.go:122] "RemoveContainer" containerID="44f99c5e7a252e1eda50ec32a45e8db42a7316551690f12808b107f881ded1b9"
Oct 30 13:53:13.663136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:13.663103    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:14.439874 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:14.439840    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:14.440894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:14.440866    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:53:17.570809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.570770    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"]
Oct 30 13:53:17.578419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.578389    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:17.593068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.593032    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"]
Oct 30 13:53:17.598615 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.598591    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-319179\""
Oct 30 13:53:17.652371 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.652329    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"]
Oct 30 13:53:17.657524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.657496    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:17.669389 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.669350    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"]
Oct 30 13:53:17.680732 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.673856    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-319179-dockercfg-c9qsh\""
Oct 30 13:53:17.772659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.766130    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d703d9f4-f5c2-4266-a584-83de96a71a23-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-319179-775886b666-7vtvz\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:17.772659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.766261    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e89ec9-a212-4721-8724-77dea9a5bab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:17.783668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.778708    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z6nn\" (UniqueName: \"kubernetes.io/projected/22e89ec9-a212-4721-8724-77dea9a5bab7-kube-api-access-5z6nn\") pod \"payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:17.908840 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.883949    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmmnb\" (UniqueName: \"kubernetes.io/projected/d703d9f4-f5c2-4266-a584-83de96a71a23-kube-api-access-mmmnb\") pod \"payload-processing-e2e-inf-b-319179-775886b666-7vtvz\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:17.986704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.986110    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e89ec9-a212-4721-8724-77dea9a5bab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:17.986704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.986236    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z6nn\" (UniqueName: \"kubernetes.io/projected/22e89ec9-a212-4721-8724-77dea9a5bab7-kube-api-access-5z6nn\") pod \"payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:17.986704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.986349    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmmnb\" (UniqueName: \"kubernetes.io/projected/d703d9f4-f5c2-4266-a584-83de96a71a23-kube-api-access-mmmnb\") pod \"payload-processing-e2e-inf-b-319179-775886b666-7vtvz\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:17.986704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.986550    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d703d9f4-f5c2-4266-a584-83de96a71a23-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-319179-775886b666-7vtvz\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:17.997750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.987726    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d703d9f4-f5c2-4266-a584-83de96a71a23-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-319179-775886b666-7vtvz\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:17.997750 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:17.989141    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e89ec9-a212-4721-8724-77dea9a5bab7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:18.030367 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.030339    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmmnb\" (UniqueName: \"kubernetes.io/projected/d703d9f4-f5c2-4266-a584-83de96a71a23-kube-api-access-mmmnb\") pod \"payload-processing-e2e-inf-b-319179-775886b666-7vtvz\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") " pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:18.032264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.032234    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z6nn\" (UniqueName: \"kubernetes.io/projected/22e89ec9-a212-4721-8724-77dea9a5bab7-kube-api-access-5z6nn\") pod \"payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:18.102411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.102369    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"]
Oct 30 13:53:18.107293 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.107269    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.124059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.123980    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:18.124059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.124039    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:53:18.128785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.128756    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"]
Oct 30 13:53:18.135980 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.135948    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:53:18.136235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.135994    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:53:18.190165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.190081    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.190165 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.190121    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrwht\" (UniqueName: \"kubernetes.io/projected/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-kube-api-access-hrwht\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.190362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.190204    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.197720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.197685    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:18.280041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.279964    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:18.293328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.291518    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.293328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.291574    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrwht\" (UniqueName: \"kubernetes.io/projected/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-kube-api-access-hrwht\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.293328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.291672    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.293328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.292425    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.304156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.304114    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.312442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.312413    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrwht\" (UniqueName: \"kubernetes.io/projected/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-kube-api-access-hrwht\") pod \"maas-api-revoke-sub-b3e2c17b-xqzw6\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.428283 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.428250    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:18.529220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.529188    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"]
Oct 30 13:53:18.530445 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:53:18.530419    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd703d9f4_f5c2_4266_a584_83de96a71a23.slice/crio-7efc91f39bdaeb4e19c78e3954d2aae9eb5dfa05f61e2f0e280976c557f0add9 WatchSource:0}: Error finding container 7efc91f39bdaeb4e19c78e3954d2aae9eb5dfa05f61e2f0e280976c557f0add9: Status 404 returned error can't find the container with id 7efc91f39bdaeb4e19c78e3954d2aae9eb5dfa05f61e2f0e280976c557f0add9
Oct 30 13:53:18.633309 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:53:18.633274    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22e89ec9_a212_4721_8724_77dea9a5bab7.slice/crio-5712ac42b895a609e0fbd1b3deeb163ae8c04be6ed5fecc53a5b52015b8baba5 WatchSource:0}: Error finding container 5712ac42b895a609e0fbd1b3deeb163ae8c04be6ed5fecc53a5b52015b8baba5: Status 404 returned error can't find the container with id 5712ac42b895a609e0fbd1b3deeb163ae8c04be6ed5fecc53a5b52015b8baba5
Oct 30 13:53:18.633756 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.633371    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"]
Oct 30 13:53:18.746785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:18.746751    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"]
Oct 30 13:53:18.747968 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:53:18.747928    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3b2b08c_bd30_4f32_8baf_bd2c46e5d57f.slice/crio-6e00a9e250d47b8d5f2649bae28dc457c116cb0c14aa96eadd7121a34ac687d9 WatchSource:0}: Error finding container 6e00a9e250d47b8d5f2649bae28dc457c116cb0c14aa96eadd7121a34ac687d9: Status 404 returned error can't find the container with id 6e00a9e250d47b8d5f2649bae28dc457c116cb0c14aa96eadd7121a34ac687d9
Oct 30 13:53:19.486737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.486614    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" event={"ID":"22e89ec9-a212-4721-8724-77dea9a5bab7","Type":"ContainerStarted","Data":"4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735"}
Oct 30 13:53:19.486737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.486706    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:19.486737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.486726    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" event={"ID":"22e89ec9-a212-4721-8724-77dea9a5bab7","Type":"ContainerStarted","Data":"5712ac42b895a609e0fbd1b3deeb163ae8c04be6ed5fecc53a5b52015b8baba5"}
Oct 30 13:53:19.488411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.488385    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz" event={"ID":"d703d9f4-f5c2-4266-a584-83de96a71a23","Type":"ContainerStarted","Data":"ed59dfb20f6a2fb6d913125796532f60d2acd9e2151c6655a8edddaccee3fc97"}
Oct 30 13:53:19.488546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.488425    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz" event={"ID":"d703d9f4-f5c2-4266-a584-83de96a71a23","Type":"ContainerStarted","Data":"7efc91f39bdaeb4e19c78e3954d2aae9eb5dfa05f61e2f0e280976c557f0add9"}
Oct 30 13:53:19.488546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.488508    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:19.490097 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.490074    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6" event={"ID":"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f","Type":"ContainerStarted","Data":"61faebd9c7ebeb1d9da6e6bb49f6feb8de2d456546d29f467566ee46a138c2b8"}
Oct 30 13:53:19.490209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.490110    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6" event={"ID":"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f","Type":"ContainerStarted","Data":"6e00a9e250d47b8d5f2649bae28dc457c116cb0c14aa96eadd7121a34ac687d9"}
Oct 30 13:53:19.525516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.525469    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" podStartSLOduration=2.525456623 podStartE2EDuration="2.525456623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:19.523702031 +0000 UTC m=+3009.783940265" watchObservedRunningTime="2026-10-30 13:53:19.525456623 +0000 UTC m=+3009.785694857"
Oct 30 13:53:19.561120 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.561066    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz" podStartSLOduration=2.561051164 podStartE2EDuration="2.561051164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:19.56000895 +0000 UTC m=+3009.820247185" watchObservedRunningTime="2026-10-30 13:53:19.561051164 +0000 UTC m=+3009.821289398"
Oct 30 13:53:19.600405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:19.600354    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6" podStartSLOduration=1.600338295 podStartE2EDuration="1.600338295s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:19.596666867 +0000 UTC m=+3009.856905102" watchObservedRunningTime="2026-10-30 13:53:19.600338295 +0000 UTC m=+3009.860576529"
Oct 30 13:53:20.499470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:20.499435    4291 generic.go:356] "Generic (PLEG): container finished" podID="d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" containerID="61faebd9c7ebeb1d9da6e6bb49f6feb8de2d456546d29f467566ee46a138c2b8" exitCode=0
Oct 30 13:53:20.499982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:20.499515    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6" event={"ID":"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f","Type":"ContainerDied","Data":"61faebd9c7ebeb1d9da6e6bb49f6feb8de2d456546d29f467566ee46a138c2b8"}
Oct 30 13:53:21.709122 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.709094    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:21.775661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.775556    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-xks-ca\") pod \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") "
Oct 30 13:53:21.775841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.775722    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-kube-api-access-hrwht\" (UniqueName: \"kubernetes.io/projected/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-kube-api-access-hrwht\") pod \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") "
Oct 30 13:53:21.775920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.775849    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-openshift-ca\") pod \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\" (UID: \"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f\") "
Oct 30 13:53:21.776199 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.776176    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-openshift-ca" pod "d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" (UID: "d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:21.776537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.776522    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:21.778267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.778234    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-xks-ca" pod "d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" (UID: "d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:21.778396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.778346    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-kube-api-access-hrwht" pod "d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" (UID: "d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f"). InnerVolumeSpecName "kube-api-access-hrwht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:21.878398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.878365    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:21.878398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:21.878396    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrwht\" (UniqueName: \"kubernetes.io/projected/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f-kube-api-access-hrwht\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:22.516778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:22.516743    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"
Oct 30 13:53:22.516979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:22.516745    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6" event={"ID":"d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f","Type":"ContainerDied","Data":"6e00a9e250d47b8d5f2649bae28dc457c116cb0c14aa96eadd7121a34ac687d9"}
Oct 30 13:53:22.516979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:22.516871    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e00a9e250d47b8d5f2649bae28dc457c116cb0c14aa96eadd7121a34ac687d9"
Oct 30 13:53:28.821235 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.821196    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"]
Oct 30 13:53:28.821985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.821969    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" containerName="revoke-keys"
Oct 30 13:53:28.821985 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.821985    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" containerName="revoke-keys"
Oct 30 13:53:28.822113 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.822104    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" containerName="revoke-keys"
Oct 30 13:53:28.826032 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.826015    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:28.842274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.842242    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"]
Oct 30 13:53:28.848179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.848153    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:53:28.848360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.848195    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:53:28.848428 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.848214    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:53:28.851864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.851843    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:53:28.990542 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.990500    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:28.990758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.990738    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:28.990817 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:28.990777    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjb4z\" (UniqueName: \"kubernetes.io/projected/13c61c25-6e3a-4fb6-a3af-6c5e11286491-kube-api-access-mjb4z\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.092842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.092755    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.092842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.092801    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjb4z\" (UniqueName: \"kubernetes.io/projected/13c61c25-6e3a-4fb6-a3af-6c5e11286491-kube-api-access-mjb4z\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.093088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.092932    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.093566 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.093540    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.095191 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.095170    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.112708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.112682    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjb4z\" (UniqueName: \"kubernetes.io/projected/13c61c25-6e3a-4fb6-a3af-6c5e11286491-kube-api-access-mjb4z\") pod \"maas-api-revoke-sub-3372c12a-n7sgf\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.142044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.142014    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:29.452894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.452861    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"]
Oct 30 13:53:29.454575 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:53:29.454548    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c61c25_6e3a_4fb6_a3af_6c5e11286491.slice/crio-d144a49109f0d5fa9c85c194251db20044711dff516d034068e0c0d2e0ec0fcd WatchSource:0}: Error finding container d144a49109f0d5fa9c85c194251db20044711dff516d034068e0c0d2e0ec0fcd: Status 404 returned error can't find the container with id d144a49109f0d5fa9c85c194251db20044711dff516d034068e0c0d2e0ec0fcd
Oct 30 13:53:29.572763 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:29.572727    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf" event={"ID":"13c61c25-6e3a-4fb6-a3af-6c5e11286491","Type":"ContainerStarted","Data":"d144a49109f0d5fa9c85c194251db20044711dff516d034068e0c0d2e0ec0fcd"}
Oct 30 13:53:30.502851 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:30.502825    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 13:53:30.503223 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:30.502900    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 13:53:30.596018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:30.595978    4291 generic.go:356] "Generic (PLEG): container finished" podID="13c61c25-6e3a-4fb6-a3af-6c5e11286491" containerID="d28facc1bfb1ab6bff5d51a0cd9b2710d86bfe521722026c08449c9e8c40d42b" exitCode=0
Oct 30 13:53:30.596207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:30.596071    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf" event={"ID":"13c61c25-6e3a-4fb6-a3af-6c5e11286491","Type":"ContainerDied","Data":"d28facc1bfb1ab6bff5d51a0cd9b2710d86bfe521722026c08449c9e8c40d42b"}
Oct 30 13:53:31.768379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.768352    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:31.833808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.833774    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13c61c25-6e3a-4fb6-a3af-6c5e11286491-kube-api-access-mjb4z\" (UniqueName: \"kubernetes.io/projected/13c61c25-6e3a-4fb6-a3af-6c5e11286491-kube-api-access-mjb4z\") pod \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") "
Oct 30 13:53:31.834056 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.834039    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-openshift-ca\") pod \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") "
Oct 30 13:53:31.834156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.834087    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-xks-ca\") pod \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\" (UID: \"13c61c25-6e3a-4fb6-a3af-6c5e11286491\") "
Oct 30 13:53:31.834489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.834453    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-openshift-ca" pod "13c61c25-6e3a-4fb6-a3af-6c5e11286491" (UID: "13c61c25-6e3a-4fb6-a3af-6c5e11286491"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:31.834733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.834715    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:31.836552 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.836526    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c61c25-6e3a-4fb6-a3af-6c5e11286491-kube-api-access-mjb4z" pod "13c61c25-6e3a-4fb6-a3af-6c5e11286491" (UID: "13c61c25-6e3a-4fb6-a3af-6c5e11286491"). InnerVolumeSpecName "kube-api-access-mjb4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:31.836876 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.836855    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-xks-ca" pod "13c61c25-6e3a-4fb6-a3af-6c5e11286491" (UID: "13c61c25-6e3a-4fb6-a3af-6c5e11286491"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:53:31.936483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.936450    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13c61c25-6e3a-4fb6-a3af-6c5e11286491-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:31.936483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:31.936482    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjb4z\" (UniqueName: \"kubernetes.io/projected/13c61c25-6e3a-4fb6-a3af-6c5e11286491-kube-api-access-mjb4z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:32.613848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:32.613812    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"
Oct 30 13:53:32.613848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:32.613829    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf" event={"ID":"13c61c25-6e3a-4fb6-a3af-6c5e11286491","Type":"ContainerDied","Data":"d144a49109f0d5fa9c85c194251db20044711dff516d034068e0c0d2e0ec0fcd"}
Oct 30 13:53:32.614125 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:32.613876    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d144a49109f0d5fa9c85c194251db20044711dff516d034068e0c0d2e0ec0fcd"
Oct 30 13:53:38.214735 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:53:38.214685    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:53:41.188328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:41.188288    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"]
Oct 30 13:53:46.189673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.189603    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd" podUID="894e3f5c-a5d0-4960-87b2-15a7dd1078b7" containerName="payload-processing" containerID="cri-o://0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46" gracePeriod=25
Oct 30 13:53:46.490154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.490107    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:53:46.597352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.597309    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-kube-api-access-627f9\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-kube-api-access-627f9\") pod \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") "
Oct 30 13:53:46.597555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.597445    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-routing-overlay-volume\") pod \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") "
Oct 30 13:53:46.597730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.597696    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-plugins-config-volume\") pod \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") "
Oct 30 13:53:46.597839 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.597812    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-routing-overlay-volume" pod "894e3f5c-a5d0-4960-87b2-15a7dd1078b7" (UID: "894e3f5c-a5d0-4960-87b2-15a7dd1078b7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:46.597916 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.597858    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-provider-credentials\") pod \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\" (UID: \"894e3f5c-a5d0-4960-87b2-15a7dd1078b7\") "
Oct 30 13:53:46.598114 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.598090    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-plugins-config-volume" pod "894e3f5c-a5d0-4960-87b2-15a7dd1078b7" (UID: "894e3f5c-a5d0-4960-87b2-15a7dd1078b7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:53:46.598425 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.598407    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:46.598486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.598430    4291 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-routing-overlay-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:46.599904 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.599883    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-provider-credentials" pod "894e3f5c-a5d0-4960-87b2-15a7dd1078b7" (UID: "894e3f5c-a5d0-4960-87b2-15a7dd1078b7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:46.599969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.599906    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-kube-api-access-627f9" pod "894e3f5c-a5d0-4960-87b2-15a7dd1078b7" (UID: "894e3f5c-a5d0-4960-87b2-15a7dd1078b7"). InnerVolumeSpecName "kube-api-access-627f9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:53:46.700369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.700329    4291 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-provider-credentials\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:46.700369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.700365    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-627f9\" (UniqueName: \"kubernetes.io/projected/894e3f5c-a5d0-4960-87b2-15a7dd1078b7-kube-api-access-627f9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:53:46.761211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.761124    4291 generic.go:356] "Generic (PLEG): container finished" podID="894e3f5c-a5d0-4960-87b2-15a7dd1078b7" containerID="0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46" exitCode=0
Oct 30 13:53:46.761211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.761169    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"
Oct 30 13:53:46.761410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.761205    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd" event={"ID":"894e3f5c-a5d0-4960-87b2-15a7dd1078b7","Type":"ContainerDied","Data":"0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46"}
Oct 30 13:53:46.761410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.761262    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd" event={"ID":"894e3f5c-a5d0-4960-87b2-15a7dd1078b7","Type":"ContainerDied","Data":"f2d6c9e83314a00465703c9f9bf27b144c8a55b1aed0260dae01df85aa3776ea"}
Oct 30 13:53:46.761410 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.761284    4291 scope.go:122] "RemoveContainer" containerID="0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46"
Oct 30 13:53:46.773942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.773914    4291 scope.go:122] "RemoveContainer" containerID="0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46"
Oct 30 13:53:46.774272 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:53:46.774251    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46\": container with ID starting with 0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46 not found: ID does not exist" containerID="0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46"
Oct 30 13:53:46.774369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.774278    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46"} err="failed to get container status \"0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46\": rpc error: code = NotFound desc = could not find container \"0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46\": container with ID starting with 0813f771ce8a0d36f73fc2ec40bd88b0f3533d54fd5cfba8d10535fb9fd42f46 not found: ID does not exist"
Oct 30 13:53:46.819576 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.819125    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"]
Oct 30 13:53:46.828415 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:46.828380    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-bd2fed/payload-processing-external-model-e2e-extmod-a-bd2fed-6c5bnsxwd"]
Oct 30 13:53:47.891088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:47.891051    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="894e3f5c-a5d0-4960-87b2-15a7dd1078b7" path="/var/lib/kubelet/pods/894e3f5c-a5d0-4960-87b2-15a7dd1078b7/volumes"
Oct 30 13:53:55.781154 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.781109    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"]
Oct 30 13:53:55.781863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.781847    4291 cpu_manager.go:414] "containerMap: removing container" podUID="894e3f5c-a5d0-4960-87b2-15a7dd1078b7" containerName="payload-processing"
Oct 30 13:53:55.781928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.781864    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="894e3f5c-a5d0-4960-87b2-15a7dd1078b7" containerName="payload-processing"
Oct 30 13:53:55.781928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.781892    4291 cpu_manager.go:414] "containerMap: removing container" podUID="13c61c25-6e3a-4fb6-a3af-6c5e11286491" containerName="revoke-keys"
Oct 30 13:53:55.781928 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.781898    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13c61c25-6e3a-4fb6-a3af-6c5e11286491" containerName="revoke-keys"
Oct 30 13:53:55.782026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.782022    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="894e3f5c-a5d0-4960-87b2-15a7dd1078b7" containerName="payload-processing"
Oct 30 13:53:55.782065 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.782035    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="13c61c25-6e3a-4fb6-a3af-6c5e11286491" containerName="revoke-keys"
Oct 30 13:53:55.787317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.787294    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:55.805346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.805314    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"]
Oct 30 13:53:55.811401 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.811370    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-d724c8\""
Oct 30 13:53:55.889485 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.889435    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zlr95\" (UniqueName: \"kubernetes.io/projected/86c76eb8-94da-4b75-b304-399be0b1c313-kube-api-access-zlr95\") pod \"payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:55.889720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.889587    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c76eb8-94da-4b75-b304-399be0b1c313-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:55.995658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.994790    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zlr95\" (UniqueName: \"kubernetes.io/projected/86c76eb8-94da-4b75-b304-399be0b1c313-kube-api-access-zlr95\") pod \"payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:55.995658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:55.994960    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c76eb8-94da-4b75-b304-399be0b1c313-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:56.017668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:56.015077    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c76eb8-94da-4b75-b304-399be0b1c313-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:56.077716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:56.073502    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zlr95\" (UniqueName: \"kubernetes.io/projected/86c76eb8-94da-4b75-b304-399be0b1c313-kube-api-access-zlr95\") pod \"payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:56.108146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:56.107830    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:56.743170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:56.743137    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"]
Oct 30 13:53:56.743664 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:53:56.743610    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c76eb8_94da_4b75_b304_399be0b1c313.slice/crio-8c5a51cb4d9325b2bd48c9bf6a9e4d6674961523c380666d1e10bbe3556b8b33 WatchSource:0}: Error finding container 8c5a51cb4d9325b2bd48c9bf6a9e4d6674961523c380666d1e10bbe3556b8b33: Status 404 returned error can't find the container with id 8c5a51cb4d9325b2bd48c9bf6a9e4d6674961523c380666d1e10bbe3556b8b33
Oct 30 13:53:56.869301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:56.869267    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" event={"ID":"86c76eb8-94da-4b75-b304-399be0b1c313","Type":"ContainerStarted","Data":"8c5a51cb4d9325b2bd48c9bf6a9e4d6674961523c380666d1e10bbe3556b8b33"}
Oct 30 13:53:57.914680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:57.914624    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:53:57.915103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:57.914726    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" event={"ID":"86c76eb8-94da-4b75-b304-399be0b1c313","Type":"ContainerStarted","Data":"76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab"}
Oct 30 13:53:57.981379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:53:57.981310    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" podStartSLOduration=2.981290512 podStartE2EDuration="2.981290512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:53:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:53:57.979962016 +0000 UTC m=+3048.240200266" watchObservedRunningTime="2026-10-30 13:53:57.981290512 +0000 UTC m=+3048.241528749"
Oct 30 13:54:01.844172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:01.844135    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"]
Oct 30 13:54:01.844819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:01.844402    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy" containerID="cri-o://1d1b8a2f0c4579038804a015157e4b0afbcb3126fd30009d5f6601787405bc09" gracePeriod=30
Oct 30 13:54:04.495716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:04.495666    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"]
Oct 30 13:54:04.496262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:04.495957    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" containerID="cri-o://89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c" gracePeriod=30
Oct 30 13:54:06.438710 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.438666    4291 patch_prober.go:29] interesting pod/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:54:06.439141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.438732    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:54:06.989970 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.989938    4291 generic.go:356] "Generic (PLEG): container finished" podID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerID="1d1b8a2f0c4579038804a015157e4b0afbcb3126fd30009d5f6601787405bc09" exitCode=0
Oct 30 13:54:06.990155 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.990018    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" event={"ID":"83bc2ea8-1f71-4718-b322-80dcc69af66a","Type":"ContainerDied","Data":"1d1b8a2f0c4579038804a015157e4b0afbcb3126fd30009d5f6601787405bc09"}
Oct 30 13:54:06.992058 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.992030    4291 generic.go:356] "Generic (PLEG): container finished" podID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerID="ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa" exitCode=0
Oct 30 13:54:06.992208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.992093    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" event={"ID":"31a6bbcd-4039-492d-8c6b-079648353ef2","Type":"ContainerDied","Data":"ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa"}
Oct 30 13:54:06.992680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.992621    4291 scope.go:122] "RemoveContainer" containerID="ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa"
Oct 30 13:54:06.995160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.995129    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:54:06.995281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.995217    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:54:06.995281 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:06.995268    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:54:07.156690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.156297    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:54:07.252734 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.252695    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-kube-api-access-wcbl6\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-kube-api-access-wcbl6\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.252945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.252773    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-envoy\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.252945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.252812    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-credential-socket\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.252945 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.252903    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-certs\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.253160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.252964    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-data\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.253160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.253050    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83bc2ea8-1f71-4718-b322-80dcc69af66a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bc2ea8-1f71-4718-b322-80dcc69af66a-istiod-ca-cert\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.253160 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.253083    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-token\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-token\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.253326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.253206    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-socket\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.253326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.253245    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-podinfo\") pod \"83bc2ea8-1f71-4718-b322-80dcc69af66a\" (UID: \"83bc2ea8-1f71-4718-b322-80dcc69af66a\") "
Oct 30 13:54:07.253768 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.253742    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-data" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.254088 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.254074    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.254453 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.254425    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-socket" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.254722 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.254693    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-certs" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.254813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.254711    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-credential-socket" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.254910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.254885    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bc2ea8-1f71-4718-b322-80dcc69af66a-istiod-ca-cert" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:07.256766 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.256729    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-kube-api-access-wcbl6" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "kube-api-access-wcbl6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:07.257042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.257010    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-envoy" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:07.257172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.257040    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-podinfo" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:07.259163 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.259139    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-token" pod "83bc2ea8-1f71-4718-b322-80dcc69af66a" (UID: "83bc2ea8-1f71-4718-b322-80dcc69af66a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:07.355896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355854    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcbl6\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-kube-api-access-wcbl6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.355896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355896    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.356135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355914    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.356135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355928    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.356135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355944    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/83bc2ea8-1f71-4718-b322-80dcc69af66a-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.356135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355958    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.356135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355970    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/83bc2ea8-1f71-4718-b322-80dcc69af66a-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:07.356135 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:07.355983    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/83bc2ea8-1f71-4718-b322-80dcc69af66a-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:08.001861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.001827    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"
Oct 30 13:54:08.001861 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.001843    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb" event={"ID":"83bc2ea8-1f71-4718-b322-80dcc69af66a","Type":"ContainerDied","Data":"0b69509512a849652c8175f4ccae1b5bd9adb05f629b1316887018525da1aa5c"}
Oct 30 13:54:08.002394 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.001915    4291 scope.go:122] "RemoveContainer" containerID="1d1b8a2f0c4579038804a015157e4b0afbcb3126fd30009d5f6601787405bc09"
Oct 30 13:54:08.004895 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.004865    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" event={"ID":"31a6bbcd-4039-492d-8c6b-079648353ef2","Type":"ContainerStarted","Data":"533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61"}
Oct 30 13:54:08.109741 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.109704    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"]
Oct 30 13:54:08.113754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.113731    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-pbpdb"]
Oct 30 13:54:08.214760 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:54:08.214672    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:54:08.476478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.476387    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:54:08.476698 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.476675    4291 patch_prober.go:29] interesting pod/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:54:08.476762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.476724    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:54:08.925201 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:08.925164    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 13:54:09.476260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.476219    4291 patch_prober.go:29] interesting pod/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:54:09.476813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.476294    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:54:09.825624 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.825598    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:54:09.887452 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.887416    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" path="/var/lib/kubelet/pods/83bc2ea8-1f71-4718-b322-80dcc69af66a/volumes"
Oct 30 13:54:09.910273 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910240    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istiod-ca-cert\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.910537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910282    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-credential-socket\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.910537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910307    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-data\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.910537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910335    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-token\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-token\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.910775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910554    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-credential-socket" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.910775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910719    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-data" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.910775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910725    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istiod-ca-cert" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:54:09.910952 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.910930    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-envoy\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.911119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.911098    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-certs\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.911252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.911149    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-podinfo\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.911252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.911237    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-socket\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.911433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.911279    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-kube-api-access-9t4zm\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-kube-api-access-9t4zm\") pod \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\" (UID: \"ac358606-1b97-4eb2-b53a-6b731b9aa36f\") "
Oct 30 13:54:09.911433 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.911374    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-certs" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.911594 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.911570    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-socket" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.912124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.912016    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.912124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.912043    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.912124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.912058    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.912124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.912073    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.912124 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.912086    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:09.913537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.913290    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-token" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:09.913537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.913470    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-envoy" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:54:09.913777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.913753    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-podinfo" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:54:09.913962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:09.913938    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-kube-api-access-9t4zm" pod "ac358606-1b97-4eb2-b53a-6b731b9aa36f" (UID: "ac358606-1b97-4eb2-b53a-6b731b9aa36f"). InnerVolumeSpecName "kube-api-access-9t4zm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:54:10.018435 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.018396    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:10.018625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.018466    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9t4zm\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-kube-api-access-9t4zm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:10.018625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.018484    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:10.018625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.018497    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ac358606-1b97-4eb2-b53a-6b731b9aa36f-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:54:10.030318 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.030276    4291 generic.go:356] "Generic (PLEG): container finished" podID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerID="89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c" exitCode=0
Oct 30 13:54:10.030491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.030340    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" event={"ID":"ac358606-1b97-4eb2-b53a-6b731b9aa36f","Type":"ContainerDied","Data":"89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c"}
Oct 30 13:54:10.030491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.030422    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8" event={"ID":"ac358606-1b97-4eb2-b53a-6b731b9aa36f","Type":"ContainerDied","Data":"68bae72cbff2fcd15d8895e0ba3e4f38454812f53c3c75cd301e6a8877334e4f"}
Oct 30 13:54:10.030491 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.030445    4291 scope.go:122] "RemoveContainer" containerID="89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c"
Oct 30 13:54:10.030845 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.030824    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"
Oct 30 13:54:10.044513 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.044486    4291 scope.go:122] "RemoveContainer" containerID="89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c"
Oct 30 13:54:10.044906 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:54:10.044884    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c\": container with ID starting with 89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c not found: ID does not exist" containerID="89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c"
Oct 30 13:54:10.044993 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.044915    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c"} err="failed to get container status \"89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c\": rpc error: code = NotFound desc = could not find container \"89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c\": container with ID starting with 89ea57552008a47714bcd2c4321dd570cf9659a77127d920ac10acd3b59cd15c not found: ID does not exist"
Oct 30 13:54:10.088961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.088855    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"]
Oct 30 13:54:10.095718 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.095677    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-lz8z8"]
Oct 30 13:54:10.476379 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.476336    4291 patch_prober.go:29] interesting pod/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:54:10.476856 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:10.476419    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:54:11.476661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:11.476351    4291 patch_prober.go:29] interesting pod/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:54:11.476661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:11.476426    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:54:11.977602 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:11.977559    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" path="/var/lib/kubelet/pods/ac358606-1b97-4eb2-b53a-6b731b9aa36f/volumes"
Oct 30 13:54:12.486681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:12.486041    4291 patch_prober.go:29] interesting pod/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused" start-of-body=
Oct 30 13:54:12.486681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:12.486106    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.36:15021/healthz/ready\": dial tcp 10.133.1.36:15021: connect: connection refused"
Oct 30 13:54:13.403737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:13.403703    4291 scope.go:122] "RemoveContainer" containerID="cfa339140d2fe967f4ddd025af527efa21db08fb730b886ba73e6d7c42c336a9"
Oct 30 13:54:13.416423 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:13.416397    4291 scope.go:122] "RemoveContainer" containerID="264c8b57daf89855959c374736f2d47bf6fa079f1e11894e194cde2a6b8fb26b"
Oct 30 13:54:13.427044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:13.427018    4291 scope.go:122] "RemoveContainer" containerID="829082cb6aefc3b18a0179519daf08cdd7705345558bb0782e7b60f8dbf427e7"
Oct 30 13:54:13.489882 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:13.489856    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:54:13.491346 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:13.491320    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:54:13.492232 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:13.492210    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 13:54:20.842906 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:20.842867    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:54:26.034143 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:26.034102    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"]
Oct 30 13:54:26.045470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:26.045431    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-96e1e746-59snf"]
Oct 30 13:54:27.885723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:27.885687    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc2681ae-9af3-46d2-a340-cd9b07a8b246" path="/var/lib/kubelet/pods/fc2681ae-9af3-46d2-a340-cd9b07a8b246/volumes"
Oct 30 13:54:37.389716 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.389668    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"]
Oct 30 13:54:37.390896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.390870    4291 cpu_manager.go:414] "containerMap: removing container" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy"
Oct 30 13:54:37.391008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.390898    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy"
Oct 30 13:54:37.391008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.390921    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy"
Oct 30 13:54:37.391008 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.390929    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy"
Oct 30 13:54:37.391227 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.391209    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="83bc2ea8-1f71-4718-b322-80dcc69af66a" containerName="istio-proxy"
Oct 30 13:54:37.391297 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.391234    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac358606-1b97-4eb2-b53a-6b731b9aa36f" containerName="istio-proxy"
Oct 30 13:54:37.399767 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.399734    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.415133 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.415099    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"]
Oct 30 13:54:37.422068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.422016    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-09014b\"/\"default-dockercfg-67lhn\""
Oct 30 13:54:37.422068 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.422023    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-09014b\"/\"openshift-service-ca.crt\""
Oct 30 13:54:37.429022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.428985    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-09014b\"/\"kube-root-ca.crt\""
Oct 30 13:54:37.429022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.428985    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-09014b\"/\"test-model-09014b-kserve-self-signed-certs\""
Oct 30 13:54:37.499418 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.499381    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-tmp-dir\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.499612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.499450    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-dshm\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.499740 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.499712    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85vz\" (UniqueName: \"kubernetes.io/projected/e68c3640-a5a8-4641-ab24-27c7f347b239-kube-api-access-k85vz\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.499825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.499760    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-model-cache\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.499927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.499906    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-home\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.500063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.500048    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68c3640-a5a8-4641-ab24-27c7f347b239-tls-certs\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.603286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.603128    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-tmp-dir\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.603286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.603214    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-dshm\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.603650 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.603389    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85vz\" (UniqueName: \"kubernetes.io/projected/e68c3640-a5a8-4641-ab24-27c7f347b239-kube-api-access-k85vz\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.603650 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.603423    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-model-cache\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.603650 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.603498    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-home\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.603650 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.603620    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68c3640-a5a8-4641-ab24-27c7f347b239-tls-certs\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.627668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.620703    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-tmp-dir\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.627668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.625054    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-model-cache\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.627668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.625346    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-home\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.628406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.628161    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68c3640-a5a8-4641-ab24-27c7f347b239-tls-certs\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.660128 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.659334    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-dshm\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.679588 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.679548    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85vz\" (UniqueName: \"kubernetes.io/projected/e68c3640-a5a8-4641-ab24-27c7f347b239-kube-api-access-k85vz\") pod \"test-model-09014b-kserve-79c89fd947-r8q2k\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") " pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:37.726276 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:37.726181    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:38.113017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:38.112987    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"]
Oct 30 13:54:38.114140 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:54:38.114112    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode68c3640_a5a8_4641_ab24_27c7f347b239.slice/crio-4d5f430be02d71c59e02831a9b2aa82c0dc504179aea4cc73c03f4b4d1992892 WatchSource:0}: Error finding container 4d5f430be02d71c59e02831a9b2aa82c0dc504179aea4cc73c03f4b4d1992892: Status 404 returned error can't find the container with id 4d5f430be02d71c59e02831a9b2aa82c0dc504179aea4cc73c03f4b4d1992892
Oct 30 13:54:38.216102 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:54:38.215812    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:54:38.359871 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:38.359832    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k" event={"ID":"e68c3640-a5a8-4641-ab24-27c7f347b239","Type":"ContainerStarted","Data":"ba0b1650a741a93d394f6958c217836604a71ed29e3f023928e8cc7ead3005d3"}
Oct 30 13:54:38.360105 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:38.359880    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k" event={"ID":"e68c3640-a5a8-4641-ab24-27c7f347b239","Type":"ContainerStarted","Data":"4d5f430be02d71c59e02831a9b2aa82c0dc504179aea4cc73c03f4b4d1992892"}
Oct 30 13:54:38.400708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:38.400653    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k" podStartSLOduration=1.400625832 podStartE2EDuration="1.400625832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:54:38.396758359 +0000 UTC m=+3088.656996594" watchObservedRunningTime="2026-10-30 13:54:38.400625832 +0000 UTC m=+3088.660864066"
Oct 30 13:54:38.726825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:38.726734    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:47.726382 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:47.726341    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:47.739019 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:47.738993    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:48.041824 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:48.041737    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"]
Oct 30 13:54:48.045598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:48.045574    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1704930-j7tjn"]
Oct 30 13:54:48.467025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:48.466994    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 13:54:49.884587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:49.884551    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ddab16a-250d-4d79-98c0-75a9828a7604" path="/var/lib/kubelet/pods/0ddab16a-250d-4d79-98c0-75a9828a7604/volumes"
Oct 30 13:54:58.032864 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:58.032778    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"]
Oct 30 13:54:58.044141 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:58.044115    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-31af5609-xvncb"]
Oct 30 13:54:59.881089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:54:59.881054    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ee0849-4bdc-466b-ac19-fbecb61152a2" path="/var/lib/kubelet/pods/06ee0849-4bdc-466b-ac19-fbecb61152a2/volumes"
Oct 30 13:55:08.216676 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:55:08.216596    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:55:13.499104 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:13.499065    4291 scope.go:122] "RemoveContainer" containerID="12fd99b10d266618d954cfa0cee887954d567b0d79b92fcd4285de7b38f4e4ef"
Oct 30 13:55:15.481793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:15.481763    4291 patch_prober.go:29] interesting pod/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 13:55:15.482261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:15.481826    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 13:55:16.788663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:16.784327    4291 generic.go:356] "Generic (PLEG): container finished" podID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerID="264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d" exitCode=0
Oct 30 13:55:16.788663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:16.784482    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" event={"ID":"769a0ac3-b411-46d3-894c-0855bbeab4a6","Type":"ContainerDied","Data":"264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d"}
Oct 30 13:55:16.788663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:16.785048    4291 scope.go:122] "RemoveContainer" containerID="264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d"
Oct 30 13:55:16.788663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:16.787880    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:55:16.788663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:16.787967    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:55:16.788663 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:16.788011    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:55:17.795117 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:17.795078    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" event={"ID":"769a0ac3-b411-46d3-894c-0855bbeab4a6","Type":"ContainerStarted","Data":"b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2"}
Oct 30 13:55:18.032432 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:18.032398    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"]
Oct 30 13:55:18.036146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:18.036123    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-60e2635a-mrxsv"]
Oct 30 13:55:18.657301 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:18.657264    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:55:18.657720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:18.657680    4291 patch_prober.go:29] interesting pod/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:55:18.657948 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:18.657913    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:55:19.663662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:19.659187    4291 patch_prober.go:29] interesting pod/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:55:19.663662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:19.659259    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:55:19.890873 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:19.890841    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="497a939d-f8d4-4607-8758-c0c3e3c116ca" path="/var/lib/kubelet/pods/497a939d-f8d4-4607-8758-c0c3e3c116ca/volumes"
Oct 30 13:55:20.664661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:20.662081    4291 patch_prober.go:29] interesting pod/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:55:20.664661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:20.662142    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:55:21.668659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:21.668602    4291 patch_prober.go:29] interesting pod/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:55:21.669188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:21.668694    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:55:22.673660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:22.673086    4291 patch_prober.go:29] interesting pod/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 13:55:22.673660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:22.673166    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 13:55:23.662842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:23.662811    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:55:23.663323 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:23.663302    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:55:23.664109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:23.664090    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 13:55:28.039522 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:28.039487    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"]
Oct 30 13:55:28.051037 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:28.051010    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c68ff24a-h9zbl"]
Oct 30 13:55:29.881070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:29.881031    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e" path="/var/lib/kubelet/pods/f25dd2fc-a9aa-4f84-b32e-dfc7e0790e1e/volumes"
Oct 30 13:55:38.211677 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:55:38.211608    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:55:40.597590 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:40.597550    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"]
Oct 30 13:55:40.598052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:40.597804    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" containerID="cri-o://a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5" gracePeriod=30
Oct 30 13:55:42.034893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:42.034860    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"]
Oct 30 13:55:42.046103 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:42.046075    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3250ea76-hjbmq"]
Oct 30 13:55:43.461790 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:43.461752    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 13:55:43.462395 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:43.461836    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 13:55:43.882207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:43.882168    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f5a33f6-467a-4519-b629-34bba8a7c261" path="/var/lib/kubelet/pods/5f5a33f6-467a-4519-b629-34bba8a7c261/volumes"
Oct 30 13:55:45.892654 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.892615    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:55:45.995962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.995924    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-certs\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.995976    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-envoy\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996101    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-podinfo\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996125    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-credential-socket\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996145    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5467869-5fad-4417-9f4f-d09c3c496e0d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467869-5fad-4417-9f4f-d09c3c496e0d-istiod-ca-cert\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996182 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996173    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-socket\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996267    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-certs" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:45.996466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996318    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-token\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-token\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996414    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-kube-api-access-rtmpk\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-kube-api-access-rtmpk\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996485    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-data\") pod \"b5467869-5fad-4417-9f4f-d09c3c496e0d\" (UID: \"b5467869-5fad-4417-9f4f-d09c3c496e0d\") "
Oct 30 13:55:45.996627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996485    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-credential-socket" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:45.996627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996573    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-socket" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:45.996627 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996584    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5467869-5fad-4417-9f4f-d09c3c496e0d-istiod-ca-cert" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:45.997556 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.996897    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-data" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:45.997695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.997608    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:45.997695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.997624    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b5467869-5fad-4417-9f4f-d09c3c496e0d-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:45.997695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.997678    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:45.997695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.997688    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:45.997924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.997699    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:45.998769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.998743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-podinfo" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:45.998881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.998809    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-envoy" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:45.998881 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.998837    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-token" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:45.999340 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:45.999325    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-kube-api-access-rtmpk" pod "b5467869-5fad-4417-9f4f-d09c3c496e0d" (UID: "b5467869-5fad-4417-9f4f-d09c3c496e0d"). InnerVolumeSpecName "kube-api-access-rtmpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:46.095489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.089875    4291 generic.go:356] "Generic (PLEG): container finished" podID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerID="a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5" exitCode=0
Oct 30 13:55:46.095489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.089933    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" event={"ID":"b5467869-5fad-4417-9f4f-d09c3c496e0d","Type":"ContainerDied","Data":"a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5"}
Oct 30 13:55:46.095489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.089968    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"
Oct 30 13:55:46.095489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.089984    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc" event={"ID":"b5467869-5fad-4417-9f4f-d09c3c496e0d","Type":"ContainerDied","Data":"1d7c61907a4ab08c86dce6ae659cfb3b3d7d744acb6706221889c9150eb08d33"}
Oct 30 13:55:46.095489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.090013    4291 scope.go:122] "RemoveContainer" containerID="a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5"
Oct 30 13:55:46.098962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.098916    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.098962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.098953    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rtmpk\" (UniqueName: \"kubernetes.io/projected/b5467869-5fad-4417-9f4f-d09c3c496e0d-kube-api-access-rtmpk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.098962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.098967    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.099335 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.098981    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b5467869-5fad-4417-9f4f-d09c3c496e0d-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:46.122197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.122176    4291 scope.go:122] "RemoveContainer" containerID="a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5"
Oct 30 13:55:46.122589 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:55:46.122561    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5\": container with ID starting with a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5 not found: ID does not exist" containerID="a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5"
Oct 30 13:55:46.122678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.122603    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5"} err="failed to get container status \"a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5\": rpc error: code = NotFound desc = could not find container \"a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5\": container with ID starting with a0657a4434ed3cdc78e8bb9e47333131c37bbb69173692bd104ed06e82209ec5 not found: ID does not exist"
Oct 30 13:55:46.152956 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.152917    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"]
Oct 30 13:55:46.157045 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:46.157012    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-cab6c5-openshift-default-69c76dd9c-4rmfc"]
Oct 30 13:55:47.919662 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:47.917267    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" path="/var/lib/kubelet/pods/b5467869-5fad-4417-9f4f-d09c3c496e0d/volumes"
Oct 30 13:55:48.482657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:48.474413    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:55:48.867220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:48.867187    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"]
Oct 30 13:55:48.867411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:48.867380    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy" containerID="cri-o://f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71" gracePeriod=30
Oct 30 13:55:52.152181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.152147    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"]
Oct 30 13:55:52.152937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.152920    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy"
Oct 30 13:55:52.153011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.152938    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy"
Oct 30 13:55:52.153094 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.153084    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5467869-5fad-4417-9f4f-d09c3c496e0d" containerName="istio-proxy"
Oct 30 13:55:52.158796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.158778    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.177598 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.177563    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"]
Oct 30 13:55:52.185681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.185188    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-ca0243-openshift-default-dockercfg-rcf2c\""
Oct 30 13:55:52.235384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.235350    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd4569-5128-4522-86f8-12b44c954428-istiod-ca-cert\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.235573 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.235514    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-socket\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.235705 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.235687    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-credential-socket\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.235772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.235741    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-envoy\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.235868 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.235851    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-certs\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.235953 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.235932    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-istio-token\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.235999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.235967    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5wc4\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-kube-api-access-l5wc4\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.236121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.236098    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd4569-5128-4522-86f8-12b44c954428-istio-podinfo\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.236211 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.236196    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-data\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.340693    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-credential-socket\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.340776    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-envoy\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.340832    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-certs\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.340888    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-istio-token\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.340922    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5wc4\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-kube-api-access-l5wc4\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.341002    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd4569-5128-4522-86f8-12b44c954428-istio-podinfo\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.341073    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-data\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.341119    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd4569-5128-4522-86f8-12b44c954428-istiod-ca-cert\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.341660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.341212    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-socket\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.342330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.341705    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-socket\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.342330 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.341961    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-credential-socket\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.345655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.342851    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-certs\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.345655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.343350    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-data\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.376659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.351722    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd4569-5128-4522-86f8-12b44c954428-istiod-ca-cert\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.391285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.382127    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-envoy\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.391285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.383345    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5wc4\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-kube-api-access-l5wc4\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.391747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.391718    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd4569-5128-4522-86f8-12b44c954428-istio-podinfo\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.410703 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.394398    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-istio-token\") pod \"e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") " pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.484979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.484939    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:52.888272 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:55:52.888238    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbafd4569_5128_4522_86f8_12b44c954428.slice/crio-1ff8aa9be2f60002b4481e2eea949f163f5ab3a85eb4640513d7c200cbc63c06 WatchSource:0}: Error finding container 1ff8aa9be2f60002b4481e2eea949f163f5ab3a85eb4640513d7c200cbc63c06: Status 404 returned error can't find the container with id 1ff8aa9be2f60002b4481e2eea949f163f5ab3a85eb4640513d7c200cbc63c06
Oct 30 13:55:52.888772 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.888745    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"]
Oct 30 13:55:52.892406 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.892362    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:55:52.892524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.892452    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:55:52.892524 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:52.892494    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:55:53.035402 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.035363    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"]
Oct 30 13:55:53.040686 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.040279    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42e7724e-5jwd7"]
Oct 30 13:55:53.187252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.187164    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" event={"ID":"bafd4569-5128-4522-86f8-12b44c954428","Type":"ContainerStarted","Data":"7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1"}
Oct 30 13:55:53.187252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.187222    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" event={"ID":"bafd4569-5128-4522-86f8-12b44c954428","Type":"ContainerStarted","Data":"1ff8aa9be2f60002b4481e2eea949f163f5ab3a85eb4640513d7c200cbc63c06"}
Oct 30 13:55:53.231203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.231142    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podStartSLOduration=1.231128383 podStartE2EDuration="1.231128383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:53.227621446 +0000 UTC m=+3163.487859692" watchObservedRunningTime="2026-10-30 13:55:53.231128383 +0000 UTC m=+3163.491366616"
Oct 30 13:55:53.485867 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.485774    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:53.488041 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.488009    4291 patch_prober.go:29] interesting pod/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:55:53.488183 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.488078    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 13:55:53.885336 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:53.885300    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20e0ea8d-bc7b-4c46-8f23-b48697dc9ade" path="/var/lib/kubelet/pods/20e0ea8d-bc7b-4c46-8f23-b48697dc9ade/volumes"
Oct 30 13:55:54.157479 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.157407    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:55:54.198083 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.198044    4291 generic.go:356] "Generic (PLEG): container finished" podID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerID="f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71" exitCode=0
Oct 30 13:55:54.198674 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.198126    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" event={"ID":"a8e9f923-8951-4f90-becb-fc85ff30a41b","Type":"ContainerDied","Data":"f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71"}
Oct 30 13:55:54.198782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.198724    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h" event={"ID":"a8e9f923-8951-4f90-becb-fc85ff30a41b","Type":"ContainerDied","Data":"26ed2fb2a662d4402fd1bbae194236a146909cc347373c4cfa527ed3c2460723"}
Oct 30 13:55:54.198782 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.198192    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"
Oct 30 13:55:54.198961 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.198763    4291 scope.go:122] "RemoveContainer" containerID="f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71"
Oct 30 13:55:54.214826 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.214796    4291 scope.go:122] "RemoveContainer" containerID="f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71"
Oct 30 13:55:54.215186 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:55:54.215150    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71\": container with ID starting with f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71 not found: ID does not exist" containerID="f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71"
Oct 30 13:55:54.215257 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.215200    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71"} err="failed to get container status \"f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71\": rpc error: code = NotFound desc = could not find container \"f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71\": container with ID starting with f2c5c80f72c749179efa732a84175357eab102a7091abb04ebe46b69cf0d2d71 not found: ID does not exist"
Oct 30 13:55:54.287302 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287265    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-kube-api-access-znrr5\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-kube-api-access-znrr5\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287353    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-token\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-token\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287473    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-envoy\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287509    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-certs\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287569 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287532    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-socket\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287652    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-credential-socket\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287693    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-data\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287739 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287728    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-podinfo\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287794    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8e9f923-8951-4f90-becb-fc85ff30a41b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e9f923-8951-4f90-becb-fc85ff30a41b-istiod-ca-cert\") pod \"a8e9f923-8951-4f90-becb-fc85ff30a41b\" (UID: \"a8e9f923-8951-4f90-becb-fc85ff30a41b\") "
Oct 30 13:55:54.287920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287821    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-certs" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.287920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.287868    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-socket" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.288096 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288039    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-data" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.288252 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288217    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-credential-socket" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.288537 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288513    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8e9f923-8951-4f90-becb-fc85ff30a41b-istiod-ca-cert" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:55:54.288774 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288753    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.288832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288781    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.288832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288795    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.288832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288809    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.288832 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.288823    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8e9f923-8951-4f90-becb-fc85ff30a41b-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.289841 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.289812    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-kube-api-access-znrr5" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "kube-api-access-znrr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:54.290029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.290009    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-token" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:55:54.290168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.290138    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-envoy" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:55:54.290264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.290199    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-podinfo" pod "a8e9f923-8951-4f90-becb-fc85ff30a41b" (UID: "a8e9f923-8951-4f90-becb-fc85ff30a41b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:55:54.390853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.390815    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znrr5\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-kube-api-access-znrr5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.390853 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.390853    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.390869    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.391084 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.390882    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8e9f923-8951-4f90-becb-fc85ff30a41b-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:55:54.491457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.491375    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:54.570764 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.570714    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"]
Oct 30 13:55:54.577708 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:54.577673    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-844856-openshift-default-5655c6cb69-v2f9h"]
Oct 30 13:55:55.207680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:55.207623    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:55.208477 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:55.208459    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:55:55.881051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:55.881012    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" path="/var/lib/kubelet/pods/a8e9f923-8951-4f90-becb-fc85ff30a41b/volumes"
Oct 30 13:55:58.413653 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.413598    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"]
Oct 30 13:55:58.415234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.415209    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy"
Oct 30 13:55:58.415338 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.415235    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy"
Oct 30 13:55:58.415713 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.415694    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8e9f923-8951-4f90-becb-fc85ff30a41b" containerName="istio-proxy"
Oct 30 13:55:58.422595 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.422567    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.427311 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.427280    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"]
Oct 30 13:55:58.443879 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.443847    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"]
Oct 30 13:55:58.444034 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.443955    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:58.446380 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.446356    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-ca0243\""
Oct 30 13:55:58.454243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.454213    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"]
Oct 30 13:55:58.459769 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.459746    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-ca0243-dockercfg-s9tsw\""
Oct 30 13:55:58.462413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.462384    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ffw5\" (UniqueName: \"kubernetes.io/projected/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-kube-api-access-8ffw5\") pod \"payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:58.462515 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.462461    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmbx\" (UniqueName: \"kubernetes.io/projected/47798c3a-363a-4a27-9148-ed8caf915e1a-kube-api-access-4xmbx\") pod \"payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.462720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.462695    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:58.462811 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.462786    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47798c3a-363a-4a27-9148-ed8caf915e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.576659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.573868    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ffw5\" (UniqueName: \"kubernetes.io/projected/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-kube-api-access-8ffw5\") pod \"payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:58.576659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.573974    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmbx\" (UniqueName: \"kubernetes.io/projected/47798c3a-363a-4a27-9148-ed8caf915e1a-kube-api-access-4xmbx\") pod \"payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.576659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.574104    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:58.576659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.574182    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47798c3a-363a-4a27-9148-ed8caf915e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.580670 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.577584    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:58.585695 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.582730    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47798c3a-363a-4a27-9148-ed8caf915e1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.619659 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.618743    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmbx\" (UniqueName: \"kubernetes.io/projected/47798c3a-363a-4a27-9148-ed8caf915e1a-kube-api-access-4xmbx\") pod \"payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.643657 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.634288    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ffw5\" (UniqueName: \"kubernetes.io/projected/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-kube-api-access-8ffw5\") pod \"payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") " pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:58.759044 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.758953    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:58.762886 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:58.762855    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:59.092368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.092340    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"]
Oct 30 13:55:59.093213 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:55:59.093188    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47798c3a_363a_4a27_9148_ed8caf915e1a.slice/crio-4356bb04d6b6beefe9af2ef62c6cbfcdf76faeebdec4969aed65a0852149fc1d WatchSource:0}: Error finding container 4356bb04d6b6beefe9af2ef62c6cbfcdf76faeebdec4969aed65a0852149fc1d: Status 404 returned error can't find the container with id 4356bb04d6b6beefe9af2ef62c6cbfcdf76faeebdec4969aed65a0852149fc1d
Oct 30 13:55:59.110695 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:55:59.110666    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3e9026a_cb2f_45b9_ae82_aaf7dad1075d.slice/crio-f55ceb8d38f2a06b255d53551e80b16566da8a1235eab364da15dbb77a856260 WatchSource:0}: Error finding container f55ceb8d38f2a06b255d53551e80b16566da8a1235eab364da15dbb77a856260: Status 404 returned error can't find the container with id f55ceb8d38f2a06b255d53551e80b16566da8a1235eab364da15dbb77a856260
Oct 30 13:55:59.111495 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.111474    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"]
Oct 30 13:55:59.270387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.270357    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd" event={"ID":"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d","Type":"ContainerStarted","Data":"32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9"}
Oct 30 13:55:59.270504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.270417    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd" event={"ID":"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d","Type":"ContainerStarted","Data":"f55ceb8d38f2a06b255d53551e80b16566da8a1235eab364da15dbb77a856260"}
Oct 30 13:55:59.270504 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.270446    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:55:59.272262 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.272235    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" event={"ID":"47798c3a-363a-4a27-9148-ed8caf915e1a","Type":"ContainerStarted","Data":"8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a"}
Oct 30 13:55:59.272372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.272288    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" event={"ID":"47798c3a-363a-4a27-9148-ed8caf915e1a","Type":"ContainerStarted","Data":"4356bb04d6b6beefe9af2ef62c6cbfcdf76faeebdec4969aed65a0852149fc1d"}
Oct 30 13:55:59.272372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.272321    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:55:59.311736 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.311693    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd" podStartSLOduration=1.311679571 podStartE2EDuration="1.311679571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:59.307280136 +0000 UTC m=+3169.567518372" watchObservedRunningTime="2026-10-30 13:55:59.311679571 +0000 UTC m=+3169.571917804"
Oct 30 13:55:59.351264 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:55:59.351215    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" podStartSLOduration=1.351202646 podStartE2EDuration="1.351202646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:55:59.347547935 +0000 UTC m=+3169.607786173" watchObservedRunningTime="2026-10-30 13:55:59.351202646 +0000 UTC m=+3169.611440879"
Oct 30 13:56:08.212803 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:56:08.212760    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:56:10.283261 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:56:10.283228    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 13:56:10.283788 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:56:10.283310    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 13:56:13.538053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:56:13.538015    4291 scope.go:122] "RemoveContainer" containerID="57a14e8f7d6ddff335e0d91000bbfb00bcc8c355842ca8fd414933e75b87e2c3"
Oct 30 13:56:13.548626 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:56:13.548602    4291 scope.go:122] "RemoveContainer" containerID="ddf9115ddf2f0fb4a83b6eff2037de4de680488b4dca1741c146b6a9e5834339"
Oct 30 13:56:13.558343 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:56:13.558321    4291 scope.go:122] "RemoveContainer" containerID="3286737e3d9e9a2f312cdd46d90b64d3f28dbc12971c5ce9ba6c4ba28e3d6b48"
Oct 30 13:56:38.212904 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:56:38.212821    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:56:43.189808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:56:43.189769    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:57:08.211937 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:57:08.211897    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:57:11.175196 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.175156    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"]
Oct 30 13:57:11.182417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.182373    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.196072 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.196043    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"]
Oct 30 13:57:11.199807 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.199786    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:11.199934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.199912    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:11.200039 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.200017    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:11.200157 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.200105    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:57:11.242920 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.242888    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.243093 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.243020    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.243230 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.243210    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dhht\" (UniqueName: \"kubernetes.io/projected/a039de63-693b-43c6-bfb5-f7ab327d4be1-kube-api-access-6dhht\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.344285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.344255    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.344478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.344332    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.344478 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.344461    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dhht\" (UniqueName: \"kubernetes.io/projected/a039de63-693b-43c6-bfb5-f7ab327d4be1-kube-api-access-6dhht\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.345077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.345049    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.346642 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.346610    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.364017 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.363992    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dhht\" (UniqueName: \"kubernetes.io/projected/a039de63-693b-43c6-bfb5-f7ab327d4be1-kube-api-access-6dhht\") pod \"maas-api-revoke-sub-fb1a3074-pw9w5\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.505852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.505779    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:11.808372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.808344    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"]
Oct 30 13:57:11.809606 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:57:11.809575    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda039de63_693b_43c6_bfb5_f7ab327d4be1.slice/crio-787a9accbfa5a91a8ca76889aa5ac63384212a5ee32edebca262d7f1a5ee6c42 WatchSource:0}: Error finding container 787a9accbfa5a91a8ca76889aa5ac63384212a5ee32edebca262d7f1a5ee6c42: Status 404 returned error can't find the container with id 787a9accbfa5a91a8ca76889aa5ac63384212a5ee32edebca262d7f1a5ee6c42
Oct 30 13:57:11.866847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:11.866814    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5" event={"ID":"a039de63-693b-43c6-bfb5-f7ab327d4be1","Type":"ContainerStarted","Data":"787a9accbfa5a91a8ca76889aa5ac63384212a5ee32edebca262d7f1a5ee6c42"}
Oct 30 13:57:12.876431 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:12.876395    4291 generic.go:356] "Generic (PLEG): container finished" podID="a039de63-693b-43c6-bfb5-f7ab327d4be1" containerID="a026d68e599449e59f0ab87c7826e84f121cf3f428d5f64662a49819416d215b" exitCode=0
Oct 30 13:57:12.876877 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:12.876471    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5" event={"ID":"a039de63-693b-43c6-bfb5-f7ab327d4be1","Type":"ContainerDied","Data":"a026d68e599449e59f0ab87c7826e84f121cf3f428d5f64662a49819416d215b"}
Oct 30 13:57:13.605664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:13.605620    4291 scope.go:122] "RemoveContainer" containerID="82893d111fe07da040b9b81fc87fc2edbf601a4fe78bf8041d8a34e127585a7a"
Oct 30 13:57:13.616802 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:13.616777    4291 scope.go:122] "RemoveContainer" containerID="6a1feba40acf0d765ed16e653805f7d4f72bd27831d2b9d29c3bc4ee250c0a2b"
Oct 30 13:57:13.626843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:13.626822    4291 scope.go:122] "RemoveContainer" containerID="f0561a1b2d5991a6d4e2f32d94f5fec0d40ea4af2c3ffd04c04bff6afd4214c0"
Oct 30 13:57:13.637026 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:13.637003    4291 scope.go:122] "RemoveContainer" containerID="525ef135775adf8238a9b4f1ad4f6919d0e884672f2ae648f07a999029de0cff"
Oct 30 13:57:14.060391 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.060364    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:14.089854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.089824    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a039de63-693b-43c6-bfb5-f7ab327d4be1-kube-api-access-6dhht\" (UniqueName: \"kubernetes.io/projected/a039de63-693b-43c6-bfb5-f7ab327d4be1-kube-api-access-6dhht\") pod \"a039de63-693b-43c6-bfb5-f7ab327d4be1\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") "
Oct 30 13:57:14.090052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.090030    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-xks-ca\") pod \"a039de63-693b-43c6-bfb5-f7ab327d4be1\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") "
Oct 30 13:57:14.090127 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.090115    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-openshift-ca\") pod \"a039de63-693b-43c6-bfb5-f7ab327d4be1\" (UID: \"a039de63-693b-43c6-bfb5-f7ab327d4be1\") "
Oct 30 13:57:14.090422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.090399    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-openshift-ca" pod "a039de63-693b-43c6-bfb5-f7ab327d4be1" (UID: "a039de63-693b-43c6-bfb5-f7ab327d4be1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:14.090737 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.090717    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:14.092042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.092012    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a039de63-693b-43c6-bfb5-f7ab327d4be1-kube-api-access-6dhht" pod "a039de63-693b-43c6-bfb5-f7ab327d4be1" (UID: "a039de63-693b-43c6-bfb5-f7ab327d4be1"). InnerVolumeSpecName "kube-api-access-6dhht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:14.092145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.092045    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-xks-ca" pod "a039de63-693b-43c6-bfb5-f7ab327d4be1" (UID: "a039de63-693b-43c6-bfb5-f7ab327d4be1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:14.192681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.192575    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dhht\" (UniqueName: \"kubernetes.io/projected/a039de63-693b-43c6-bfb5-f7ab327d4be1-kube-api-access-6dhht\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:14.192681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.192604    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a039de63-693b-43c6-bfb5-f7ab327d4be1-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:14.896678 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.896621    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5" event={"ID":"a039de63-693b-43c6-bfb5-f7ab327d4be1","Type":"ContainerDied","Data":"787a9accbfa5a91a8ca76889aa5ac63384212a5ee32edebca262d7f1a5ee6c42"}
Oct 30 13:57:14.897003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.896983    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="787a9accbfa5a91a8ca76889aa5ac63384212a5ee32edebca262d7f1a5ee6c42"
Oct 30 13:57:14.897204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:14.897190    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"
Oct 30 13:57:22.060720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.060629    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"]
Oct 30 13:57:22.603098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.603063    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"]
Oct 30 13:57:22.603828 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.603803    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a039de63-693b-43c6-bfb5-f7ab327d4be1" containerName="revoke-keys"
Oct 30 13:57:22.603896 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.603829    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a039de63-693b-43c6-bfb5-f7ab327d4be1" containerName="revoke-keys"
Oct 30 13:57:22.603969 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.603960    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="a039de63-693b-43c6-bfb5-f7ab327d4be1" containerName="revoke-keys"
Oct 30 13:57:22.607345 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.607330    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.622715 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.622690    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:22.625621 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.625592    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"]
Oct 30 13:57:22.631067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.631034    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:22.631067 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.631053    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:57:22.631234 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.631150    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:22.744808 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.744766    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.745010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.744821    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twjzf\" (UniqueName: \"kubernetes.io/projected/d63e9d83-6131-421c-9588-b68fd33e9128-kube-api-access-twjzf\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.745010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.744938    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.846210 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.846159    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.846413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.846238    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twjzf\" (UniqueName: \"kubernetes.io/projected/d63e9d83-6131-421c-9588-b68fd33e9128-kube-api-access-twjzf\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.846413 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.846288    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.846992 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.846970    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.848476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.848456    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-xks-ca\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.864688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.864614    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twjzf\" (UniqueName: \"kubernetes.io/projected/d63e9d83-6131-421c-9588-b68fd33e9128-kube-api-access-twjzf\") pod \"maas-api-revoke-sub-776c2587-rb8sr\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:22.922911 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:22.922874    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:23.268220 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:23.268185    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"]
Oct 30 13:57:23.269483 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:57:23.269452    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd63e9d83_6131_421c_9588_b68fd33e9128.slice/crio-28ed2b9c0f4882709a37ba729ffc56c3563dba70451ffaad7ed93c8244b9f80d WatchSource:0}: Error finding container 28ed2b9c0f4882709a37ba729ffc56c3563dba70451ffaad7ed93c8244b9f80d: Status 404 returned error can't find the container with id 28ed2b9c0f4882709a37ba729ffc56c3563dba70451ffaad7ed93c8244b9f80d
Oct 30 13:57:23.969218 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:23.969174    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr" event={"ID":"d63e9d83-6131-421c-9588-b68fd33e9128","Type":"ContainerStarted","Data":"5bd46f446dba6a6cfc68602d43997ca9c556b58011ce2801ca9747ccd770e2b1"}
Oct 30 13:57:23.969445 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:23.969287    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr" event={"ID":"d63e9d83-6131-421c-9588-b68fd33e9128","Type":"ContainerStarted","Data":"28ed2b9c0f4882709a37ba729ffc56c3563dba70451ffaad7ed93c8244b9f80d"}
Oct 30 13:57:24.008536 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:24.008477    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr" podStartSLOduration=2.008465017 podStartE2EDuration="2.008465017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:57:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:57:24.007922273 +0000 UTC m=+3254.268160507" watchObservedRunningTime="2026-10-30 13:57:24.008465017 +0000 UTC m=+3254.268703252"
Oct 30 13:57:24.979132 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:24.979097    4291 generic.go:356] "Generic (PLEG): container finished" podID="d63e9d83-6131-421c-9588-b68fd33e9128" containerID="5bd46f446dba6a6cfc68602d43997ca9c556b58011ce2801ca9747ccd770e2b1" exitCode=0
Oct 30 13:57:24.979523 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:24.979174    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr" event={"ID":"d63e9d83-6131-421c-9588-b68fd33e9128","Type":"ContainerDied","Data":"5bd46f446dba6a6cfc68602d43997ca9c556b58011ce2801ca9747ccd770e2b1"}
Oct 30 13:57:26.151712 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.151679    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:26.302847 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.302756    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-openshift-ca\") pod \"d63e9d83-6131-421c-9588-b68fd33e9128\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") "
Oct 30 13:57:26.303010 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.302971    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d63e9d83-6131-421c-9588-b68fd33e9128-kube-api-access-twjzf\" (UniqueName: \"kubernetes.io/projected/d63e9d83-6131-421c-9588-b68fd33e9128-kube-api-access-twjzf\") pod \"d63e9d83-6131-421c-9588-b68fd33e9128\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") "
Oct 30 13:57:26.303063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.303032    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-openshift-ca" pod "d63e9d83-6131-421c-9588-b68fd33e9128" (UID: "d63e9d83-6131-421c-9588-b68fd33e9128"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:26.303063 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.303050    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-xks-ca\") pod \"d63e9d83-6131-421c-9588-b68fd33e9128\" (UID: \"d63e9d83-6131-421c-9588-b68fd33e9128\") "
Oct 30 13:57:26.303398 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.303382    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:26.305102 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.305081    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-xks-ca" pod "d63e9d83-6131-421c-9588-b68fd33e9128" (UID: "d63e9d83-6131-421c-9588-b68fd33e9128"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:26.305204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.305086    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d63e9d83-6131-421c-9588-b68fd33e9128-kube-api-access-twjzf" pod "d63e9d83-6131-421c-9588-b68fd33e9128" (UID: "d63e9d83-6131-421c-9588-b68fd33e9128"). InnerVolumeSpecName "kube-api-access-twjzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:26.404528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.404485    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twjzf\" (UniqueName: \"kubernetes.io/projected/d63e9d83-6131-421c-9588-b68fd33e9128-kube-api-access-twjzf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:26.404528 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.404519    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d63e9d83-6131-421c-9588-b68fd33e9128-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:26.994869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.994833    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr" event={"ID":"d63e9d83-6131-421c-9588-b68fd33e9128","Type":"ContainerDied","Data":"28ed2b9c0f4882709a37ba729ffc56c3563dba70451ffaad7ed93c8244b9f80d"}
Oct 30 13:57:26.994869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.994864    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"
Oct 30 13:57:26.995111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:26.994890    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ed2b9c0f4882709a37ba729ffc56c3563dba70451ffaad7ed93c8244b9f80d"
Oct 30 13:57:37.083983 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.083908    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv" podUID="f7eaa41c-2d83-46d3-a360-904c3b46d0b9" containerName="main" containerID="cri-o://9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898" gracePeriod=45
Oct 30 13:57:37.419419 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.419390    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:57:37.607195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.607158    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tmp-dir\") pod \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") "
Oct 30 13:57:37.607424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.607347    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-home\") pod \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") "
Oct 30 13:57:37.607533 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.607499    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tmp-dir" pod "f7eaa41c-2d83-46d3-a360-904c3b46d0b9" (UID: "f7eaa41c-2d83-46d3-a360-904c3b46d0b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:37.607668 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.607626    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-home" pod "f7eaa41c-2d83-46d3-a360-904c3b46d0b9" (UID: "f7eaa41c-2d83-46d3-a360-904c3b46d0b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:37.607770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.607716    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-dshm\") pod \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") "
Oct 30 13:57:37.607770 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.607750    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tls-certs\") pod \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") "
Oct 30 13:57:37.607944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.607908    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-model-cache\") pod \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") "
Oct 30 13:57:37.608025 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.608009    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-kube-api-access-xdr7f\" (UniqueName: \"kubernetes.io/projected/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-kube-api-access-xdr7f\") pod \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\" (UID: \"f7eaa41c-2d83-46d3-a360-904c3b46d0b9\") "
Oct 30 13:57:37.608303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.608275    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-model-cache" pod "f7eaa41c-2d83-46d3-a360-904c3b46d0b9" (UID: "f7eaa41c-2d83-46d3-a360-904c3b46d0b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:37.609249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.609224    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:37.609424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.609409    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:37.609577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.609562    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:37.610326 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.610305    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-dshm" pod "f7eaa41c-2d83-46d3-a360-904c3b46d0b9" (UID: "f7eaa41c-2d83-46d3-a360-904c3b46d0b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:57:37.610545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.610526    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tls-certs" pod "f7eaa41c-2d83-46d3-a360-904c3b46d0b9" (UID: "f7eaa41c-2d83-46d3-a360-904c3b46d0b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:37.610775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.610752    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-kube-api-access-xdr7f" pod "f7eaa41c-2d83-46d3-a360-904c3b46d0b9" (UID: "f7eaa41c-2d83-46d3-a360-904c3b46d0b9"). InnerVolumeSpecName "kube-api-access-xdr7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:37.710707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.710551    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdr7f\" (UniqueName: \"kubernetes.io/projected/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-kube-api-access-xdr7f\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:37.710707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.710591    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:37.710707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:37.710607    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7eaa41c-2d83-46d3-a360-904c3b46d0b9-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:38.101929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.101883    4291 generic.go:356] "Generic (PLEG): container finished" podID="f7eaa41c-2d83-46d3-a360-904c3b46d0b9" containerID="9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898" exitCode=0
Oct 30 13:57:38.102464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.101944    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"
Oct 30 13:57:38.102464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.101957    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv" event={"ID":"f7eaa41c-2d83-46d3-a360-904c3b46d0b9","Type":"ContainerDied","Data":"9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898"}
Oct 30 13:57:38.102464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.102011    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv" event={"ID":"f7eaa41c-2d83-46d3-a360-904c3b46d0b9","Type":"ContainerDied","Data":"f884dbb8aa95727496151d500dd6634775769f2e2edaf2a49c4aecd9e536f455"}
Oct 30 13:57:38.102464 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.102042    4291 scope.go:122] "RemoveContainer" containerID="9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898"
Oct 30 13:57:38.113186 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.113159    4291 scope.go:122] "RemoveContainer" containerID="9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898"
Oct 30 13:57:38.113579 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:57:38.113549    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898\": container with ID starting with 9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898 not found: ID does not exist" containerID="9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898"
Oct 30 13:57:38.113707 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.113594    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898"} err="failed to get container status \"9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898\": rpc error: code = NotFound desc = could not find container \"9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898\": container with ID starting with 9be181a086cafdefb62668f649dba0a9f5d038d16460042c1895781968bcc898 not found: ID does not exist"
Oct 30 13:57:38.182655 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.181967    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"]
Oct 30 13:57:38.186316 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:38.186286    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-5494c9/test-model-5494c9-kserve-7df459fbdc-m8hpv"]
Oct 30 13:57:38.214279 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:57:38.214246    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:57:39.887434 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:39.887398    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7eaa41c-2d83-46d3-a360-904c3b46d0b9" path="/var/lib/kubelet/pods/f7eaa41c-2d83-46d3-a360-904c3b46d0b9/volumes"
Oct 30 13:57:44.967538 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.967506    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"]
Oct 30 13:57:44.968225 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.968210    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d63e9d83-6131-421c-9588-b68fd33e9128" containerName="revoke-keys"
Oct 30 13:57:44.968275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.968227    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d63e9d83-6131-421c-9588-b68fd33e9128" containerName="revoke-keys"
Oct 30 13:57:44.968275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.968256    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f7eaa41c-2d83-46d3-a360-904c3b46d0b9" containerName="main"
Oct 30 13:57:44.968275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.968260    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7eaa41c-2d83-46d3-a360-904c3b46d0b9" containerName="main"
Oct 30 13:57:44.968387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.968378    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d63e9d83-6131-421c-9588-b68fd33e9128" containerName="revoke-keys"
Oct 30 13:57:44.968424 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.968394    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7eaa41c-2d83-46d3-a360-904c3b46d0b9" containerName="main"
Oct 30 13:57:44.972022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.972005    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:44.992937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.992904    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"]
Oct 30 13:57:44.998794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.998767    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:57:44.999181 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:44.999125    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:57:45.020303 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.020277    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:57:45.020527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.020364    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:57:45.049915 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.049886    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.050079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.049932    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.050152 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.050078    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl98x\" (UniqueName: \"kubernetes.io/projected/aecd3c24-34ab-4597-8142-d032b25036c5-kube-api-access-bl98x\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.151834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.151787    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.160924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.160894    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl98x\" (UniqueName: \"kubernetes.io/projected/aecd3c24-34ab-4597-8142-d032b25036c5-kube-api-access-bl98x\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.161442 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.161412    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.162393 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.162372    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.170156 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.170128    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.181200 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.181177    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl98x\" (UniqueName: \"kubernetes.io/projected/aecd3c24-34ab-4597-8142-d032b25036c5-kube-api-access-bl98x\") pod \"maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.291299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.291234    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:45.600693 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:45.600667    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"]
Oct 30 13:57:45.602575 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:57:45.602546    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaecd3c24_34ab_4597_8142_d032b25036c5.slice/crio-330104b4d22c4455a4df6f9917b555e99a48a43a8d2204a65bd37b163278efb6 WatchSource:0}: Error finding container 330104b4d22c4455a4df6f9917b555e99a48a43a8d2204a65bd37b163278efb6: Status 404 returned error can't find the container with id 330104b4d22c4455a4df6f9917b555e99a48a43a8d2204a65bd37b163278efb6
Oct 30 13:57:46.179449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.179421    4291 generic.go:356] "Generic (PLEG): container finished" podID="bafd4569-5128-4522-86f8-12b44c954428" containerID="7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1" exitCode=0
Oct 30 13:57:46.179844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.179496    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" event={"ID":"bafd4569-5128-4522-86f8-12b44c954428","Type":"ContainerDied","Data":"7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1"}
Oct 30 13:57:46.179999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.179980    4291 scope.go:122] "RemoveContainer" containerID="7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1"
Oct 30 13:57:46.181299 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.181276    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp" event={"ID":"aecd3c24-34ab-4597-8142-d032b25036c5","Type":"ContainerStarted","Data":"906da1402bc7fbecee17d5654705cb575fb9c1c1d11369c789c785e250adf7ab"}
Oct 30 13:57:46.181396 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.181321    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp" event={"ID":"aecd3c24-34ab-4597-8142-d032b25036c5","Type":"ContainerStarted","Data":"330104b4d22c4455a4df6f9917b555e99a48a43a8d2204a65bd37b163278efb6"}
Oct 30 13:57:46.182054 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.182028    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:57:46.182111 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.182100    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:57:46.182149 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:46.182137    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 13:57:47.191844 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:47.191804    4291 generic.go:356] "Generic (PLEG): container finished" podID="aecd3c24-34ab-4597-8142-d032b25036c5" containerID="906da1402bc7fbecee17d5654705cb575fb9c1c1d11369c789c785e250adf7ab" exitCode=0
Oct 30 13:57:47.192253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:47.191881    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp" event={"ID":"aecd3c24-34ab-4597-8142-d032b25036c5","Type":"ContainerDied","Data":"906da1402bc7fbecee17d5654705cb575fb9c1c1d11369c789c785e250adf7ab"}
Oct 30 13:57:47.194059 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:47.194033    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" event={"ID":"bafd4569-5128-4522-86f8-12b44c954428","Type":"ContainerStarted","Data":"dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0"}
Oct 30 13:57:47.485458 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:47.485359    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:57:47.485625 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:47.485584    4291 patch_prober.go:29] interesting pod/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:57:47.485696 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:47.485667    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 13:57:48.414028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.413999    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:48.486029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.485939    4291 patch_prober.go:29] interesting pod/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:57:48.486029 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.486001    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 13:57:48.513178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.513140    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-xks-ca\") pod \"aecd3c24-34ab-4597-8142-d032b25036c5\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") "
Oct 30 13:57:48.513374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.513197    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-openshift-ca\") pod \"aecd3c24-34ab-4597-8142-d032b25036c5\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") "
Oct 30 13:57:48.513374 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.513274    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aecd3c24-34ab-4597-8142-d032b25036c5-kube-api-access-bl98x\" (UniqueName: \"kubernetes.io/projected/aecd3c24-34ab-4597-8142-d032b25036c5-kube-api-access-bl98x\") pod \"aecd3c24-34ab-4597-8142-d032b25036c5\" (UID: \"aecd3c24-34ab-4597-8142-d032b25036c5\") "
Oct 30 13:57:48.513714 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.513687    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-openshift-ca" pod "aecd3c24-34ab-4597-8142-d032b25036c5" (UID: "aecd3c24-34ab-4597-8142-d032b25036c5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:57:48.514852 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.514824    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:48.516139 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.516052    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-xks-ca" pod "aecd3c24-34ab-4597-8142-d032b25036c5" (UID: "aecd3c24-34ab-4597-8142-d032b25036c5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:57:48.516288 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.516206    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aecd3c24-34ab-4597-8142-d032b25036c5-kube-api-access-bl98x" pod "aecd3c24-34ab-4597-8142-d032b25036c5" (UID: "aecd3c24-34ab-4597-8142-d032b25036c5"). InnerVolumeSpecName "kube-api-access-bl98x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:57:48.616372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.616337    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aecd3c24-34ab-4597-8142-d032b25036c5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:48.616372 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:48.616370    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bl98x\" (UniqueName: \"kubernetes.io/projected/aecd3c24-34ab-4597-8142-d032b25036c5-kube-api-access-bl98x\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:57:49.209042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:49.209015    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"
Oct 30 13:57:49.209042 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:49.209038    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp" event={"ID":"aecd3c24-34ab-4597-8142-d032b25036c5","Type":"ContainerDied","Data":"330104b4d22c4455a4df6f9917b555e99a48a43a8d2204a65bd37b163278efb6"}
Oct 30 13:57:49.209286 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:49.209087    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="330104b4d22c4455a4df6f9917b555e99a48a43a8d2204a65bd37b163278efb6"
Oct 30 13:57:49.485680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:49.485571    4291 patch_prober.go:29] interesting pod/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:57:49.485680 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:49.485618    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 13:57:50.488090 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:50.488047    4291 patch_prober.go:29] interesting pod/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:57:50.488589 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:50.488114    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 13:57:51.485731 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:51.485697    4291 patch_prober.go:29] interesting pod/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused" start-of-body=
Oct 30 13:57:51.485910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:51.485745    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.46:15021/healthz/ready\": dial tcp 10.133.1.46:15021: connect: connection refused"
Oct 30 13:57:52.490725 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:52.490693    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:57:52.491168 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:52.491154    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:57:52.491778 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:52.491754    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 13:57:55.401449 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:57:55.401412    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"]
Oct 30 13:58:00.401924 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:00.401865    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" podUID="c82b7230-22c5-476d-8452-1c7f301e5630" containerName="payload-processing" containerID="cri-o://b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0" gracePeriod=25
Oct 30 13:58:01.719138 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.719104    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"]
Oct 30 13:58:01.719907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.719887    4291 cpu_manager.go:414] "containerMap: removing container" podUID="aecd3c24-34ab-4597-8142-d032b25036c5" containerName="revoke-keys"
Oct 30 13:58:01.719907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.719905    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aecd3c24-34ab-4597-8142-d032b25036c5" containerName="revoke-keys"
Oct 30 13:58:01.720051 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.720029    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="aecd3c24-34ab-4597-8142-d032b25036c5" containerName="revoke-keys"
Oct 30 13:58:01.722858 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.722843    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.741030 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.740999    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"]
Oct 30 13:58:01.748386 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.748357    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:58:01.748738 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.748705    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:58:01.748930 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.748915    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:58:01.750489 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.750468    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:58:01.810144 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.810107    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.810373 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.810352    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxb5z\" (UniqueName: \"kubernetes.io/projected/dbc54b30-fd04-4968-beb4-2b021bd92264-kube-api-access-sxb5z\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.810422 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.810408    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.911941 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.911904    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxb5z\" (UniqueName: \"kubernetes.io/projected/dbc54b30-fd04-4968-beb4-2b021bd92264-kube-api-access-sxb5z\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.912153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.911986    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.912153 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.912112    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.912758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.912728    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.914383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.914355    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:01.931274 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:01.931249    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxb5z\" (UniqueName: \"kubernetes.io/projected/dbc54b30-fd04-4968-beb4-2b021bd92264-kube-api-access-sxb5z\") pod \"maas-api-revoke-sub-d6551a7f-df52j\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:02.035921 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:02.035828    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:02.348877 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:58:02.348846    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc54b30_fd04_4968_beb4_2b021bd92264.slice/crio-ced78d328caad81d0a4632ab99c9c40e773d1c1290328edcd55796807778e1d7 WatchSource:0}: Error finding container ced78d328caad81d0a4632ab99c9c40e773d1c1290328edcd55796807778e1d7: Status 404 returned error can't find the container with id ced78d328caad81d0a4632ab99c9c40e773d1c1290328edcd55796807778e1d7
Oct 30 13:58:02.350240 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:02.350214    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"]
Oct 30 13:58:03.316792 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:03.316754    4291 generic.go:356] "Generic (PLEG): container finished" podID="dbc54b30-fd04-4968-beb4-2b021bd92264" containerID="77e85a0ac8e852eff1b9303a3ba4a60e4c4102f7fa01f7eece18ffaf5a8233a4" exitCode=0
Oct 30 13:58:03.317203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:03.316836    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j" event={"ID":"dbc54b30-fd04-4968-beb4-2b021bd92264","Type":"ContainerDied","Data":"77e85a0ac8e852eff1b9303a3ba4a60e4c4102f7fa01f7eece18ffaf5a8233a4"}
Oct 30 13:58:03.317203 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:03.316904    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j" event={"ID":"dbc54b30-fd04-4968-beb4-2b021bd92264","Type":"ContainerStarted","Data":"ced78d328caad81d0a4632ab99c9c40e773d1c1290328edcd55796807778e1d7"}
Oct 30 13:58:04.203907 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.203867    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" podUID="c82b7230-22c5-476d-8452-1c7f301e5630" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:58:04.496915 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.496892    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:04.569999 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.569962    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-openshift-ca\") pod \"dbc54b30-fd04-4968-beb4-2b021bd92264\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") "
Oct 30 13:58:04.570193 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.570060    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-xks-ca\") pod \"dbc54b30-fd04-4968-beb4-2b021bd92264\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") "
Oct 30 13:58:04.570285 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.570196    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc54b30-fd04-4968-beb4-2b021bd92264-kube-api-access-sxb5z\" (UniqueName: \"kubernetes.io/projected/dbc54b30-fd04-4968-beb4-2b021bd92264-kube-api-access-sxb5z\") pod \"dbc54b30-fd04-4968-beb4-2b021bd92264\" (UID: \"dbc54b30-fd04-4968-beb4-2b021bd92264\") "
Oct 30 13:58:04.570436 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.570406    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-openshift-ca" pod "dbc54b30-fd04-4968-beb4-2b021bd92264" (UID: "dbc54b30-fd04-4968-beb4-2b021bd92264"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:04.570937 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.570911    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.572378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.572353    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-xks-ca" pod "dbc54b30-fd04-4968-beb4-2b021bd92264" (UID: "dbc54b30-fd04-4968-beb4-2b021bd92264"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:04.572944 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.572921    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc54b30-fd04-4968-beb4-2b021bd92264-kube-api-access-sxb5z" pod "dbc54b30-fd04-4968-beb4-2b021bd92264" (UID: "dbc54b30-fd04-4968-beb4-2b021bd92264"). InnerVolumeSpecName "kube-api-access-sxb5z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:04.671942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.671914    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dbc54b30-fd04-4968-beb4-2b021bd92264-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:04.671942 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:04.671941    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxb5z\" (UniqueName: \"kubernetes.io/projected/dbc54b30-fd04-4968-beb4-2b021bd92264-kube-api-access-sxb5z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:05.333039 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:05.333009    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"
Oct 30 13:58:05.333231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:05.333048    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j" event={"ID":"dbc54b30-fd04-4968-beb4-2b021bd92264","Type":"ContainerDied","Data":"ced78d328caad81d0a4632ab99c9c40e773d1c1290328edcd55796807778e1d7"}
Oct 30 13:58:05.333231 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:05.333099    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ced78d328caad81d0a4632ab99c9c40e773d1c1290328edcd55796807778e1d7"
Oct 30 13:58:08.223658 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:58:08.223203    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:58:14.204331 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:14.204291    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" podUID="c82b7230-22c5-476d-8452-1c7f301e5630" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:58:17.217465 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.217432    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"]
Oct 30 13:58:17.218296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.218277    4291 cpu_manager.go:414] "containerMap: removing container" podUID="dbc54b30-fd04-4968-beb4-2b021bd92264" containerName="revoke-keys"
Oct 30 13:58:17.218357 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.218299    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc54b30-fd04-4968-beb4-2b021bd92264" containerName="revoke-keys"
Oct 30 13:58:17.218411 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.218402    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc54b30-fd04-4968-beb4-2b021bd92264" containerName="revoke-keys"
Oct 30 13:58:17.222842 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.222823    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.238854 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.238829    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:58:17.250403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.250379    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:58:17.250403 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.250386    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:58:17.250559 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.250451    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:58:17.255540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.255515    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"]
Oct 30 13:58:17.303405 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.303378    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.303546 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.303420    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqb5n\" (UniqueName: \"kubernetes.io/projected/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-kube-api-access-nqb5n\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.303905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.303615    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.405553 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.405514    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.405796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.405728    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.405796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.405754    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqb5n\" (UniqueName: \"kubernetes.io/projected/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-kube-api-access-nqb5n\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.406397 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.406373    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.408022 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.408005    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.429404 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.429378    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqb5n\" (UniqueName: \"kubernetes.io/projected/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-kube-api-access-nqb5n\") pod \"maas-api-revoke-sub-68e64745-mq7qm\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.536838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.536756    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:17.847003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:17.846974    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"]
Oct 30 13:58:17.847686 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:58:17.847661    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d9c1728_a4ae_424d_a941_6a3e4e4a406b.slice/crio-e201000ad9d9e0e93d8f76ff23a8ddbf2c71a09dd7a7cc880143a5c7ff09bf33 WatchSource:0}: Error finding container e201000ad9d9e0e93d8f76ff23a8ddbf2c71a09dd7a7cc880143a5c7ff09bf33: Status 404 returned error can't find the container with id e201000ad9d9e0e93d8f76ff23a8ddbf2c71a09dd7a7cc880143a5c7ff09bf33
Oct 30 13:58:18.429747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:18.429701    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm" event={"ID":"0d9c1728-a4ae-424d-a941-6a3e4e4a406b","Type":"ContainerStarted","Data":"37d0d3313d51ecffb5c9b0f270417605c689b7a91704563f69f58c13f7f9bf9c"}
Oct 30 13:58:18.430115 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:18.429761    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm" event={"ID":"0d9c1728-a4ae-424d-a941-6a3e4e4a406b","Type":"ContainerStarted","Data":"e201000ad9d9e0e93d8f76ff23a8ddbf2c71a09dd7a7cc880143a5c7ff09bf33"}
Oct 30 13:58:18.481172 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:18.481129    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm" podStartSLOduration=1.481117015 podStartE2EDuration="1.481117015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:18.4769195 +0000 UTC m=+3308.737157761" watchObservedRunningTime="2026-10-30 13:58:18.481117015 +0000 UTC m=+3308.741355270"
Oct 30 13:58:19.440217 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:19.440185    4291 generic.go:356] "Generic (PLEG): container finished" podID="0d9c1728-a4ae-424d-a941-6a3e4e4a406b" containerID="37d0d3313d51ecffb5c9b0f270417605c689b7a91704563f69f58c13f7f9bf9c" exitCode=0
Oct 30 13:58:19.440647 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:19.440263    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm" event={"ID":"0d9c1728-a4ae-424d-a941-6a3e4e4a406b","Type":"ContainerDied","Data":"37d0d3313d51ecffb5c9b0f270417605c689b7a91704563f69f58c13f7f9bf9c"}
Oct 30 13:58:20.683835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.683744    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:20.745809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.745782    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:58:20.856893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.856859    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c82b7230-22c5-476d-8452-1c7f301e5630-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b7230-22c5-476d-8452-1c7f301e5630-plugins-config-volume\") pod \"c82b7230-22c5-476d-8452-1c7f301e5630\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") "
Oct 30 13:58:20.857061 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857045    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-openshift-ca\") pod \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") "
Oct 30 13:58:20.857108 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857084    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-kube-api-access-nqb5n\" (UniqueName: \"kubernetes.io/projected/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-kube-api-access-nqb5n\") pod \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") "
Oct 30 13:58:20.857146 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857121    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c82b7230-22c5-476d-8452-1c7f301e5630-kube-api-access-nrff7\" (UniqueName: \"kubernetes.io/projected/c82b7230-22c5-476d-8452-1c7f301e5630-kube-api-access-nrff7\") pod \"c82b7230-22c5-476d-8452-1c7f301e5630\" (UID: \"c82b7230-22c5-476d-8452-1c7f301e5630\") "
Oct 30 13:58:20.857237 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857221    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-xks-ca\") pod \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\" (UID: \"0d9c1728-a4ae-424d-a941-6a3e4e4a406b\") "
Oct 30 13:58:20.857349 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857222    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c82b7230-22c5-476d-8452-1c7f301e5630-plugins-config-volume" pod "c82b7230-22c5-476d-8452-1c7f301e5630" (UID: "c82b7230-22c5-476d-8452-1c7f301e5630"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:20.857460 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857436    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-openshift-ca" pod "0d9c1728-a4ae-424d-a941-6a3e4e4a406b" (UID: "0d9c1728-a4ae-424d-a941-6a3e4e4a406b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:20.857929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857904    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c82b7230-22c5-476d-8452-1c7f301e5630-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:20.857929 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.857926    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:20.859516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.859486    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-xks-ca" pod "0d9c1728-a4ae-424d-a941-6a3e4e4a406b" (UID: "0d9c1728-a4ae-424d-a941-6a3e4e4a406b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:20.859516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.859503    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c82b7230-22c5-476d-8452-1c7f301e5630-kube-api-access-nrff7" pod "c82b7230-22c5-476d-8452-1c7f301e5630" (UID: "c82b7230-22c5-476d-8452-1c7f301e5630"). InnerVolumeSpecName "kube-api-access-nrff7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:20.859516 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.859488    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-kube-api-access-nqb5n" pod "0d9c1728-a4ae-424d-a941-6a3e4e4a406b" (UID: "0d9c1728-a4ae-424d-a941-6a3e4e4a406b"). InnerVolumeSpecName "kube-api-access-nqb5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:20.959878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.959775    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqb5n\" (UniqueName: \"kubernetes.io/projected/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-kube-api-access-nqb5n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:20.959878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.959805    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrff7\" (UniqueName: \"kubernetes.io/projected/c82b7230-22c5-476d-8452-1c7f301e5630-kube-api-access-nrff7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:20.959878 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:20.959824    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0d9c1728-a4ae-424d-a941-6a3e4e4a406b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:21.493890 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.493860    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"
Oct 30 13:58:21.494077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.493863    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm" event={"ID":"0d9c1728-a4ae-424d-a941-6a3e4e4a406b","Type":"ContainerDied","Data":"e201000ad9d9e0e93d8f76ff23a8ddbf2c71a09dd7a7cc880143a5c7ff09bf33"}
Oct 30 13:58:21.494077 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.493991    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e201000ad9d9e0e93d8f76ff23a8ddbf2c71a09dd7a7cc880143a5c7ff09bf33"
Oct 30 13:58:21.495697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.495670    4291 generic.go:356] "Generic (PLEG): container finished" podID="c82b7230-22c5-476d-8452-1c7f301e5630" containerID="b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0" exitCode=0
Oct 30 13:58:21.495697 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.495685    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"
Oct 30 13:58:21.495905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.495724    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" event={"ID":"c82b7230-22c5-476d-8452-1c7f301e5630","Type":"ContainerDied","Data":"b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0"}
Oct 30 13:58:21.495905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.495775    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n" event={"ID":"c82b7230-22c5-476d-8452-1c7f301e5630","Type":"ContainerDied","Data":"54cb177aca509a661f246fbdd0af88ce9469a66d6d7ae73fbabac28cd5b00260"}
Oct 30 13:58:21.495905 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.495796    4291 scope.go:122] "RemoveContainer" containerID="b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0"
Oct 30 13:58:21.507894 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.507875    4291 scope.go:122] "RemoveContainer" containerID="b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0"
Oct 30 13:58:21.508156 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:58:21.508139    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0\": container with ID starting with b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0 not found: ID does not exist" containerID="b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0"
Oct 30 13:58:21.508204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.508162    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0"} err="failed to get container status \"b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0\": rpc error: code = NotFound desc = could not find container \"b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0\": container with ID starting with b33e76e462ad7617ad2e706e41643e2969945df39457fbcfa5de9e05899f99e0 not found: ID does not exist"
Oct 30 13:58:21.560785 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.560750    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"]
Oct 30 13:58:21.564747 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.564723    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-5494c9-6b497784c6-rb47n"]
Oct 30 13:58:21.630470 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.630437    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 13:58:21.881471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:21.881434    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c82b7230-22c5-476d-8452-1c7f301e5630" path="/var/lib/kubelet/pods/c82b7230-22c5-476d-8452-1c7f301e5630/volumes"
Oct 30 13:58:22.035810 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:22.035777    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"]
Oct 30 13:58:22.039833 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:22.039801    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b3e2c17b-xqzw6"]
Oct 30 13:58:23.882684 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:23.882630    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f" path="/var/lib/kubelet/pods/d3b2b08c-bd30-4f32-8baf-bd2c46e5d57f/volumes"
Oct 30 13:58:32.055762 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:32.055724    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"]
Oct 30 13:58:32.057347 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:32.057310    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3372c12a-n7sgf"]
Oct 30 13:58:33.881018 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:33.880980    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13c61c25-6e3a-4fb6-a3af-6c5e11286491" path="/var/lib/kubelet/pods/13c61c25-6e3a-4fb6-a3af-6c5e11286491/volumes"
Oct 30 13:58:38.214754 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:58:38.214706    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:58:39.096170 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.096134    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"]
Oct 30 13:58:39.096870 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.096855    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0d9c1728-a4ae-424d-a941-6a3e4e4a406b" containerName="revoke-keys"
Oct 30 13:58:39.096943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.096871    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d9c1728-a4ae-424d-a941-6a3e4e4a406b" containerName="revoke-keys"
Oct 30 13:58:39.096943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.096888    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c82b7230-22c5-476d-8452-1c7f301e5630" containerName="payload-processing"
Oct 30 13:58:39.096943 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.096893    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c82b7230-22c5-476d-8452-1c7f301e5630" containerName="payload-processing"
Oct 30 13:58:39.097053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.097032    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d9c1728-a4ae-424d-a941-6a3e4e4a406b" containerName="revoke-keys"
Oct 30 13:58:39.097053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.097045    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c82b7230-22c5-476d-8452-1c7f301e5630" containerName="payload-processing"
Oct 30 13:58:39.102307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.102284    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.120520 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.120489    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"]
Oct 30 13:58:39.125565 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.125538    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:58:39.132989 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.132959    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:58:39.133267 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.133242    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:58:39.133383 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.133359    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:58:39.272024 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.271975    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvb9r\" (UniqueName: \"kubernetes.io/projected/e90c9377-4653-4c09-92a5-208ffb18718f-kube-api-access-zvb9r\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.272448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.272049    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.272448 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.272199    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.374329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.374224    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvb9r\" (UniqueName: \"kubernetes.io/projected/e90c9377-4653-4c09-92a5-208ffb18718f-kube-api-access-zvb9r\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.374329 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.374304    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.374587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.374429    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.375123 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.375103    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.376875 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.376847    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.394690 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.394626    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvb9r\" (UniqueName: \"kubernetes.io/projected/e90c9377-4653-4c09-92a5-208ffb18718f-kube-api-access-zvb9r\") pod \"maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.419577 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.419543    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:39.736775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:39.736745    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"]
Oct 30 13:58:39.737149 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:58:39.737127    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode90c9377_4653_4c09_92a5_208ffb18718f.slice/crio-b0d9f6947a20167808e584c6be42bc2965d7849fdbf59c5dd3c96021c96296fa WatchSource:0}: Error finding container b0d9f6947a20167808e584c6be42bc2965d7849fdbf59c5dd3c96021c96296fa: Status 404 returned error can't find the container with id b0d9f6947a20167808e584c6be42bc2965d7849fdbf59c5dd3c96021c96296fa
Oct 30 13:58:40.740462 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:40.740421    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5" event={"ID":"e90c9377-4653-4c09-92a5-208ffb18718f","Type":"ContainerStarted","Data":"9eb6a5ad4e8d33fcb0fabef8980bf681214e75ed3b9161182f1cda56b44368c1"}
Oct 30 13:58:40.741184 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:40.741158    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5" event={"ID":"e90c9377-4653-4c09-92a5-208ffb18718f","Type":"ContainerStarted","Data":"b0d9f6947a20167808e584c6be42bc2965d7849fdbf59c5dd3c96021c96296fa"}
Oct 30 13:58:40.783605 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:40.783550    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5" podStartSLOduration=1.783534726 podStartE2EDuration="1.783534726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:40.780857249 +0000 UTC m=+3331.041095484" watchObservedRunningTime="2026-10-30 13:58:40.783534726 +0000 UTC m=+3331.043773002"
Oct 30 13:58:41.053599 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:41.053503    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"]
Oct 30 13:58:41.060298 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:41.060240    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-5494c9-vq8dp"]
Oct 30 13:58:41.750984 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:41.750947    4291 generic.go:356] "Generic (PLEG): container finished" podID="e90c9377-4653-4c09-92a5-208ffb18718f" containerID="9eb6a5ad4e8d33fcb0fabef8980bf681214e75ed3b9161182f1cda56b44368c1" exitCode=0
Oct 30 13:58:41.751390 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:41.751042    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5" event={"ID":"e90c9377-4653-4c09-92a5-208ffb18718f","Type":"ContainerDied","Data":"9eb6a5ad4e8d33fcb0fabef8980bf681214e75ed3b9161182f1cda56b44368c1"}
Oct 30 13:58:41.881167 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:41.881130    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aecd3c24-34ab-4597-8142-d032b25036c5" path="/var/lib/kubelet/pods/aecd3c24-34ab-4597-8142-d032b25036c5/volumes"
Oct 30 13:58:42.971164 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:42.971132    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:43.082988 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.082902    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e90c9377-4653-4c09-92a5-208ffb18718f-kube-api-access-zvb9r\" (UniqueName: \"kubernetes.io/projected/e90c9377-4653-4c09-92a5-208ffb18718f-kube-api-access-zvb9r\") pod \"e90c9377-4653-4c09-92a5-208ffb18718f\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") "
Oct 30 13:58:43.083171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.082997    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-xks-ca\") pod \"e90c9377-4653-4c09-92a5-208ffb18718f\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") "
Oct 30 13:58:43.083171 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.083026    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-openshift-ca\") pod \"e90c9377-4653-4c09-92a5-208ffb18718f\" (UID: \"e90c9377-4653-4c09-92a5-208ffb18718f\") "
Oct 30 13:58:43.083466 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.083439    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-openshift-ca" pod "e90c9377-4653-4c09-92a5-208ffb18718f" (UID: "e90c9377-4653-4c09-92a5-208ffb18718f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:43.083571 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.083551    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:43.085209 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.085185    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-xks-ca" pod "e90c9377-4653-4c09-92a5-208ffb18718f" (UID: "e90c9377-4653-4c09-92a5-208ffb18718f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:43.085317 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.085241    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e90c9377-4653-4c09-92a5-208ffb18718f-kube-api-access-zvb9r" pod "e90c9377-4653-4c09-92a5-208ffb18718f" (UID: "e90c9377-4653-4c09-92a5-208ffb18718f"). InnerVolumeSpecName "kube-api-access-zvb9r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:43.185314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.185280    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvb9r\" (UniqueName: \"kubernetes.io/projected/e90c9377-4653-4c09-92a5-208ffb18718f-kube-api-access-zvb9r\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:43.185314 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.185311    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e90c9377-4653-4c09-92a5-208ffb18718f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:43.810249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.810213    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"
Oct 30 13:58:43.810471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.810250    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5" event={"ID":"e90c9377-4653-4c09-92a5-208ffb18718f","Type":"ContainerDied","Data":"b0d9f6947a20167808e584c6be42bc2965d7849fdbf59c5dd3c96021c96296fa"}
Oct 30 13:58:43.810471 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:43.810309    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b0d9f6947a20167808e584c6be42bc2965d7849fdbf59c5dd3c96021c96296fa"
Oct 30 13:58:47.809369 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:47.809329    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"]
Oct 30 13:58:47.842527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:47.842484    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy" containerID="cri-o://278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102" gracePeriod=30
Oct 30 13:58:49.531178 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.531135    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"]
Oct 30 13:58:49.531909 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.531891    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e90c9377-4653-4c09-92a5-208ffb18718f" containerName="revoke-keys"
Oct 30 13:58:49.531962 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.531910    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e90c9377-4653-4c09-92a5-208ffb18718f" containerName="revoke-keys"
Oct 30 13:58:49.532046 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.532035    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e90c9377-4653-4c09-92a5-208ffb18718f" containerName="revoke-keys"
Oct 30 13:58:49.535601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.535586    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.562004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.561976    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"]
Oct 30 13:58:49.568547 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.568523    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:58:49.568720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.568565    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:58:49.568720 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.568628    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:58:49.569620 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.569603    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:58:49.710484 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.704688    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.711028 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.710988    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8cxx\" (UniqueName: \"kubernetes.io/projected/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-kube-api-access-b8cxx\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.711300 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.711276    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.813166 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.813066    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.813359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.813245    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8cxx\" (UniqueName: \"kubernetes.io/projected/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-kube-api-access-b8cxx\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.813359 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.813321    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.813789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.813770    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.815818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.815786    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.832417 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.832384    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8cxx\" (UniqueName: \"kubernetes.io/projected/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-kube-api-access-b8cxx\") pod \"maas-api-revoke-keys-e2e-inf-b-996b8b-snttz\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:49.850363 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:49.850337    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:50.003590 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:50.003547    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"]
Oct 30 13:58:50.165207 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:50.165174    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"]
Oct 30 13:58:50.167592 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:58:50.167568    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4da5e508_19a5_4bfe_94e2_88f8e689e3ba.slice/crio-6b4a517bfe0b9b8ba40b9d9a873a476917135bdfeaa67b48a06be4b53d9509b3 WatchSource:0}: Error finding container 6b4a517bfe0b9b8ba40b9d9a873a476917135bdfeaa67b48a06be4b53d9509b3: Status 404 returned error can't find the container with id 6b4a517bfe0b9b8ba40b9d9a873a476917135bdfeaa67b48a06be4b53d9509b3
Oct 30 13:58:50.869260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:50.869226    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz" event={"ID":"4da5e508-19a5-4bfe-94e2-88f8e689e3ba","Type":"ContainerStarted","Data":"1e3453ca8a88d188e46c027917e73bb11a68d434ee8f51819d50df2ea71368f6"}
Oct 30 13:58:50.869793 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:50.869307    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz" event={"ID":"4da5e508-19a5-4bfe-94e2-88f8e689e3ba","Type":"ContainerStarted","Data":"6b4a517bfe0b9b8ba40b9d9a873a476917135bdfeaa67b48a06be4b53d9509b3"}
Oct 30 13:58:50.909688 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:50.909602    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz" podStartSLOduration=1.9095886499999999 podStartE2EDuration="1.90958865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:58:50.908950487 +0000 UTC m=+3341.169188722" watchObservedRunningTime="2026-10-30 13:58:50.90958865 +0000 UTC m=+3341.169826884"
Oct 30 13:58:51.880629 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:51.880586    4291 generic.go:356] "Generic (PLEG): container finished" podID="4da5e508-19a5-4bfe-94e2-88f8e689e3ba" containerID="1e3453ca8a88d188e46c027917e73bb11a68d434ee8f51819d50df2ea71368f6" exitCode=0
Oct 30 13:58:51.883848 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:51.883798    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz" event={"ID":"4da5e508-19a5-4bfe-94e2-88f8e689e3ba","Type":"ContainerDied","Data":"1e3453ca8a88d188e46c027917e73bb11a68d434ee8f51819d50df2ea71368f6"}
Oct 30 13:58:53.067486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.067456    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:53.094260 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.094221    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-openshift-ca\") pod \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") "
Oct 30 13:58:53.094483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.094278    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-kube-api-access-b8cxx\" (UniqueName: \"kubernetes.io/projected/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-kube-api-access-b8cxx\") pod \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") "
Oct 30 13:58:53.094483 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.094342    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-xks-ca\") pod \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\" (UID: \"4da5e508-19a5-4bfe-94e2-88f8e689e3ba\") "
Oct 30 13:58:53.094672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.094626    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-openshift-ca" pod "4da5e508-19a5-4bfe-94e2-88f8e689e3ba" (UID: "4da5e508-19a5-4bfe-94e2-88f8e689e3ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:53.095205 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.095177    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.097777 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.097737    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-kube-api-access-b8cxx" pod "4da5e508-19a5-4bfe-94e2-88f8e689e3ba" (UID: "4da5e508-19a5-4bfe-94e2-88f8e689e3ba"). InnerVolumeSpecName "kube-api-access-b8cxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:53.102243 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.102215    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-xks-ca" pod "4da5e508-19a5-4bfe-94e2-88f8e689e3ba" (UID: "4da5e508-19a5-4bfe-94e2-88f8e689e3ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:58:53.146244 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.146214    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:58:53.196630 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.196588    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-socket\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.196869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.196713    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-envoy\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.196869 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.196851    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-podinfo\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.197004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.196897    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-socket" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:53.197004 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.196916    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-kube-api-access-wjkr5\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-kube-api-access-wjkr5\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.197150 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.197128    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-token\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-token\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.197294 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.197271    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-data\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.197584 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.197557    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-data" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:53.197973 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.197622    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istiod-ca-cert\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.198608 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.197945    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istiod-ca-cert" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:58:53.198784 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.198741    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-certs\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.199070 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.199049    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-certs" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:53.199195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.198774    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-credential-socket\") pod \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\" (UID: \"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801\") "
Oct 30 13:58:53.199328 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.199308    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-credential-socket" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:53.199587 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.199565    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-podinfo" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 13:58:53.199754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.199586    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-envoy" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 13:58:53.199754 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.199625    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-kube-api-access-wjkr5" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "kube-api-access-wjkr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:53.200005 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.199988    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200006    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200015    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200024    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200035    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200044    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8cxx\" (UniqueName: \"kubernetes.io/projected/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-kube-api-access-b8cxx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200059    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjkr5\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-kube-api-access-wjkr5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200068    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4da5e508-19a5-4bfe-94e2-88f8e689e3ba-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200079 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200078    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200086    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.200482 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.200022    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-token" pod "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" (UID: "a77f16eb-16b5-40cb-a1ac-2f0cc8da1801"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:58:53.322863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.321955    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:58:53.899979 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.899954    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"
Oct 30 13:58:53.900121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.899989    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz" event={"ID":"4da5e508-19a5-4bfe-94e2-88f8e689e3ba","Type":"ContainerDied","Data":"6b4a517bfe0b9b8ba40b9d9a873a476917135bdfeaa67b48a06be4b53d9509b3"}
Oct 30 13:58:53.900121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.900042    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b4a517bfe0b9b8ba40b9d9a873a476917135bdfeaa67b48a06be4b53d9509b3"
Oct 30 13:58:53.901498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.901474    4291 generic.go:356] "Generic (PLEG): container finished" podID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerID="278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102" exitCode=0
Oct 30 13:58:53.901607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.901518    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"
Oct 30 13:58:53.901607 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.901557    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" event={"ID":"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801","Type":"ContainerDied","Data":"278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102"}
Oct 30 13:58:53.901795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.901607    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l" event={"ID":"a77f16eb-16b5-40cb-a1ac-2f0cc8da1801","Type":"ContainerDied","Data":"e13d48e665803c4723157bf937370d4948e35dd1a0be6eff171b1914e5111db3"}
Oct 30 13:58:53.901795 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.901667    4291 scope.go:122] "RemoveContainer" containerID="278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102"
Oct 30 13:58:53.912958 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.912789    4291 scope.go:122] "RemoveContainer" containerID="278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102"
Oct 30 13:58:53.913075 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:58:53.913056    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102\": container with ID starting with 278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102 not found: ID does not exist" containerID="278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102"
Oct 30 13:58:53.913121 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.913085    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102"} err="failed to get container status \"278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102\": rpc error: code = NotFound desc = could not find container \"278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102\": container with ID starting with 278418619bb90bb75aeee33c8fb530af7aff43a45a4d82466c090f97bf572102 not found: ID does not exist"
Oct 30 13:58:53.969520 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.969487    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"]
Oct 30 13:58:53.975744 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:53.975716    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qng5l"]
Oct 30 13:58:55.004208 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:55.004128    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" podUID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" containerName="payload-pre-processing" containerID="cri-o://2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8" gracePeriod=25
Oct 30 13:58:55.881052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:55.881014    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" path="/var/lib/kubelet/pods/a77f16eb-16b5-40cb-a1ac-2f0cc8da1801/volumes"
Oct 30 13:58:59.438098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:59.438053    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" podUID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:58:59.914825 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:59.914787    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"]
Oct 30 13:58:59.944660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:58:59.944617    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"]
Oct 30 13:59:04.915384 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:04.915339    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" podUID="03b01c81-164a-46a8-8a40-da1c7131c10e" containerName="payload-processing" containerID="cri-o://82d16ece91e5e045d81c2ada06c0bd78977bc52c477c2c280cbdb882b6d2ab86" gracePeriod=25
Oct 30 13:59:04.945789 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:04.945746    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" podUID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" containerName="payload-pre-processing" containerID="cri-o://e08fe47be7e0049f88b315f0cd9e0fb31f1d39371c11a42ab07cc24f2220255b" gracePeriod=25
Oct 30 13:59:05.156511 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:05.156189    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"]
Oct 30 13:59:05.160872 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:05.160837    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51605e-j7hf5"]
Oct 30 13:59:05.882073 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:05.882033    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e90c9377-4653-4c09-92a5-208ffb18718f" path="/var/lib/kubelet/pods/e90c9377-4653-4c09-92a5-208ffb18718f/volumes"
Oct 30 13:59:08.213200 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:59:08.213150    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:59:08.557366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:08.557282    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" podUID="03b01c81-164a-46a8-8a40-da1c7131c10e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:59:08.557366 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:08.557284    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" podUID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:59:09.438052 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:09.438012    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" podUID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:59:14.337089 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.337045    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"]
Oct 30 13:59:14.338728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.338698    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4da5e508-19a5-4bfe-94e2-88f8e689e3ba" containerName="revoke-keys"
Oct 30 13:59:14.338728 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.338726    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4da5e508-19a5-4bfe-94e2-88f8e689e3ba" containerName="revoke-keys"
Oct 30 13:59:14.338934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.338857    4291 cpu_manager.go:414] "containerMap: removing container" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy"
Oct 30 13:59:14.338934 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.338866    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy"
Oct 30 13:59:14.339195 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.339171    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77f16eb-16b5-40cb-a1ac-2f0cc8da1801" containerName="istio-proxy"
Oct 30 13:59:14.339270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.339215    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4da5e508-19a5-4bfe-94e2-88f8e689e3ba" containerName="revoke-keys"
Oct 30 13:59:14.344661 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.344623    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.354950 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.354916    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"]
Oct 30 13:59:14.361109 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.361081    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:59:14.361296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.361095    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:59:14.361296 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.361147    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:59:14.370822 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.370795    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:59:14.443555 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.443518    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.443809 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.443754    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9ghz\" (UniqueName: \"kubernetes.io/projected/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-kube-api-access-k9ghz\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.443927 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.443912    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.545813 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.545776    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.546003 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.545955    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9ghz\" (UniqueName: \"kubernetes.io/projected/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-kube-api-access-k9ghz\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.546092 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.546076    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.546748 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.546723    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.548307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.548287    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.565673 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.565612    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9ghz\" (UniqueName: \"kubernetes.io/projected/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-kube-api-access-k9ghz\") pod \"maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.663488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.663448    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:14.974910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:14.973496    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"]
Oct 30 13:59:14.979145 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:59:14.979112    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc6b5a3_1dd6_487e_9bba_f63c844adf06.slice/crio-f8aa9b88631fe9d4e2001180d355184a152660cef6e2f5db6b5e57b16f28ebe9 WatchSource:0}: Error finding container f8aa9b88631fe9d4e2001180d355184a152660cef6e2f5db6b5e57b16f28ebe9: Status 404 returned error can't find the container with id f8aa9b88631fe9d4e2001180d355184a152660cef6e2f5db6b5e57b16f28ebe9
Oct 30 13:59:15.109476 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.109440    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7" event={"ID":"1dc6b5a3-1dd6-487e-9bba-f63c844adf06","Type":"ContainerStarted","Data":"f8aa9b88631fe9d4e2001180d355184a152660cef6e2f5db6b5e57b16f28ebe9"}
Oct 30 13:59:15.261843 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.261817    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:59:15.356891 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.356857    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47f9a857-ba8b-4c06-bd76-faa29e505dbf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f9a857-ba8b-4c06-bd76-faa29e505dbf-plugins-config-volume\") pod \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") "
Oct 30 13:59:15.357368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.357086    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f9a857-ba8b-4c06-bd76-faa29e505dbf-kube-api-access-mxz9n\" (UniqueName: \"kubernetes.io/projected/47f9a857-ba8b-4c06-bd76-faa29e505dbf-kube-api-access-mxz9n\") pod \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\" (UID: \"47f9a857-ba8b-4c06-bd76-faa29e505dbf\") "
Oct 30 13:59:15.357368 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.357272    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47f9a857-ba8b-4c06-bd76-faa29e505dbf-plugins-config-volume" pod "47f9a857-ba8b-4c06-bd76-faa29e505dbf" (UID: "47f9a857-ba8b-4c06-bd76-faa29e505dbf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:15.357819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.357796    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47f9a857-ba8b-4c06-bd76-faa29e505dbf-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:15.360430 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.360346    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f9a857-ba8b-4c06-bd76-faa29e505dbf-kube-api-access-mxz9n" pod "47f9a857-ba8b-4c06-bd76-faa29e505dbf" (UID: "47f9a857-ba8b-4c06-bd76-faa29e505dbf"). InnerVolumeSpecName "kube-api-access-mxz9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:15.465086 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:15.462223    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxz9n\" (UniqueName: \"kubernetes.io/projected/47f9a857-ba8b-4c06-bd76-faa29e505dbf-kube-api-access-mxz9n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:16.118387 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.118350    4291 generic.go:356] "Generic (PLEG): container finished" podID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" containerID="2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8" exitCode=0
Oct 30 13:59:16.118593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.118408    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"
Oct 30 13:59:16.118593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.118410    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" event={"ID":"47f9a857-ba8b-4c06-bd76-faa29e505dbf","Type":"ContainerDied","Data":"2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8"}
Oct 30 13:59:16.118593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.118541    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg" event={"ID":"47f9a857-ba8b-4c06-bd76-faa29e505dbf","Type":"ContainerDied","Data":"0cea6d1b33f2d4d73e6830031918f47a1824c64413960489e411cb979be25742"}
Oct 30 13:59:16.118593 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.118576    4291 scope.go:122] "RemoveContainer" containerID="2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8"
Oct 30 13:59:16.120492 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.120459    4291 generic.go:356] "Generic (PLEG): container finished" podID="1dc6b5a3-1dd6-487e-9bba-f63c844adf06" containerID="a200e87452b311259ec70a2fc20fa0a2857a75dd9a5eabdea15274ca92b765b8" exitCode=0
Oct 30 13:59:16.120612 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.120587    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7" event={"ID":"1dc6b5a3-1dd6-487e-9bba-f63c844adf06","Type":"ContainerDied","Data":"a200e87452b311259ec70a2fc20fa0a2857a75dd9a5eabdea15274ca92b765b8"}
Oct 30 13:59:16.130270 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.130250    4291 scope.go:122] "RemoveContainer" containerID="2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8"
Oct 30 13:59:16.130605 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:59:16.130581    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8\": container with ID starting with 2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8 not found: ID does not exist" containerID="2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8"
Oct 30 13:59:16.130677 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.130617    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8"} err="failed to get container status \"2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8\": rpc error: code = NotFound desc = could not find container \"2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8\": container with ID starting with 2ed2251675db984d1c1e2c296a0cec32859992d7a4bbc4bb07a45419f0554fd8 not found: ID does not exist"
Oct 30 13:59:16.211572 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.211460    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"]
Oct 30 13:59:16.215730 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:16.215699    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51605e-7fc9b7c7c9-qj8fg"]
Oct 30 13:59:17.334353 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.334322    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:17.389672 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.389608    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-kube-api-access-k9ghz\" (UniqueName: \"kubernetes.io/projected/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-kube-api-access-k9ghz\") pod \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") "
Oct 30 13:59:17.389912 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.389892    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-openshift-ca\") pod \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") "
Oct 30 13:59:17.390011 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.389995    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-xks-ca\") pod \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\" (UID: \"1dc6b5a3-1dd6-487e-9bba-f63c844adf06\") "
Oct 30 13:59:17.390352 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.390321    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-openshift-ca" pod "1dc6b5a3-1dd6-487e-9bba-f63c844adf06" (UID: "1dc6b5a3-1dd6-487e-9bba-f63c844adf06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:17.390796 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.390770    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:17.392414 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.392371    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-kube-api-access-k9ghz" pod "1dc6b5a3-1dd6-487e-9bba-f63c844adf06" (UID: "1dc6b5a3-1dd6-487e-9bba-f63c844adf06"). InnerVolumeSpecName "kube-api-access-k9ghz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:17.392775 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.392743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-xks-ca" pod "1dc6b5a3-1dd6-487e-9bba-f63c844adf06" (UID: "1dc6b5a3-1dd6-487e-9bba-f63c844adf06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:59:17.492053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.492019    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:17.492053 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.492049    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9ghz\" (UniqueName: \"kubernetes.io/projected/1dc6b5a3-1dd6-487e-9bba-f63c844adf06-kube-api-access-k9ghz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:17.880278 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:17.880239    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" path="/var/lib/kubelet/pods/47f9a857-ba8b-4c06-bd76-faa29e505dbf/volumes"
Oct 30 13:59:18.162306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:18.162218    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7" event={"ID":"1dc6b5a3-1dd6-487e-9bba-f63c844adf06","Type":"ContainerDied","Data":"f8aa9b88631fe9d4e2001180d355184a152660cef6e2f5db6b5e57b16f28ebe9"}
Oct 30 13:59:18.162306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:18.162285    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8aa9b88631fe9d4e2001180d355184a152660cef6e2f5db6b5e57b16f28ebe9"
Oct 30 13:59:18.162306 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:18.162234    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"
Oct 30 13:59:18.557119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:18.557039    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" podUID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:59:18.557119 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:18.557047    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" podUID="03b01c81-164a-46a8-8a40-da1c7131c10e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 13:59:25.233968 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.233933    4291 generic.go:356] "Generic (PLEG): container finished" podID="03b01c81-164a-46a8-8a40-da1c7131c10e" containerID="82d16ece91e5e045d81c2ada06c0bd78977bc52c477c2c280cbdb882b6d2ab86" exitCode=0
Oct 30 13:59:25.234488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.234003    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" event={"ID":"03b01c81-164a-46a8-8a40-da1c7131c10e","Type":"ContainerDied","Data":"82d16ece91e5e045d81c2ada06c0bd78977bc52c477c2c280cbdb882b6d2ab86"}
Oct 30 13:59:25.234488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.234056    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989" event={"ID":"03b01c81-164a-46a8-8a40-da1c7131c10e","Type":"ContainerDied","Data":"0585a56e9a4b19757bc5de5b53f27753a9c6c575021a3fbe2b00667df4ed68bf"}
Oct 30 13:59:25.234488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.234076    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0585a56e9a4b19757bc5de5b53f27753a9c6c575021a3fbe2b00667df4ed68bf"
Oct 30 13:59:25.236151 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.236128    4291 generic.go:356] "Generic (PLEG): container finished" podID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" containerID="e08fe47be7e0049f88b315f0cd9e0fb31f1d39371c11a42ab07cc24f2220255b" exitCode=0
Oct 30 13:59:25.236307 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.236198    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" event={"ID":"08bebcb8-9b91-4ee8-ae68-b6457e01f397","Type":"ContainerDied","Data":"e08fe47be7e0049f88b315f0cd9e0fb31f1d39371c11a42ab07cc24f2220255b"}
Oct 30 13:59:25.243527 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.243506    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:59:25.285702 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.284862    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:59:25.350660 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.342091    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08bebcb8-9b91-4ee8-ae68-b6457e01f397-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bebcb8-9b91-4ee8-ae68-b6457e01f397-plugins-config-volume\") pod \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") "
Oct 30 13:59:25.358658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.353034    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08bebcb8-9b91-4ee8-ae68-b6457e01f397-kube-api-access-hmth5\" (UniqueName: \"kubernetes.io/projected/08bebcb8-9b91-4ee8-ae68-b6457e01f397-kube-api-access-hmth5\") pod \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\" (UID: \"08bebcb8-9b91-4ee8-ae68-b6457e01f397\") "
Oct 30 13:59:25.358658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.353150    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03b01c81-164a-46a8-8a40-da1c7131c10e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b01c81-164a-46a8-8a40-da1c7131c10e-plugins-config-volume\") pod \"03b01c81-164a-46a8-8a40-da1c7131c10e\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") "
Oct 30 13:59:25.358658 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.353325    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03b01c81-164a-46a8-8a40-da1c7131c10e-kube-api-access-c87nm\" (UniqueName: \"kubernetes.io/projected/03b01c81-164a-46a8-8a40-da1c7131c10e-kube-api-access-c87nm\") pod \"03b01c81-164a-46a8-8a40-da1c7131c10e\" (UID: \"03b01c81-164a-46a8-8a40-da1c7131c10e\") "
Oct 30 13:59:25.367687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.367116    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03b01c81-164a-46a8-8a40-da1c7131c10e-plugins-config-volume" pod "03b01c81-164a-46a8-8a40-da1c7131c10e" (UID: "03b01c81-164a-46a8-8a40-da1c7131c10e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:25.367687 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.367683    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08bebcb8-9b91-4ee8-ae68-b6457e01f397-plugins-config-volume" pod "08bebcb8-9b91-4ee8-ae68-b6457e01f397" (UID: "08bebcb8-9b91-4ee8-ae68-b6457e01f397"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:25.477564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.380297    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03b01c81-164a-46a8-8a40-da1c7131c10e-kube-api-access-c87nm" pod "03b01c81-164a-46a8-8a40-da1c7131c10e" (UID: "03b01c81-164a-46a8-8a40-da1c7131c10e"). InnerVolumeSpecName "kube-api-access-c87nm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:25.477564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.417291    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08bebcb8-9b91-4ee8-ae68-b6457e01f397-kube-api-access-hmth5" pod "08bebcb8-9b91-4ee8-ae68-b6457e01f397" (UID: "08bebcb8-9b91-4ee8-ae68-b6457e01f397"). InnerVolumeSpecName "kube-api-access-hmth5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:25.496505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.496470    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08bebcb8-9b91-4ee8-ae68-b6457e01f397-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:25.496505 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.496503    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmth5\" (UniqueName: \"kubernetes.io/projected/08bebcb8-9b91-4ee8-ae68-b6457e01f397-kube-api-access-hmth5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:25.496758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.496518    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03b01c81-164a-46a8-8a40-da1c7131c10e-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:25.496758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:25.496531    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c87nm\" (UniqueName: \"kubernetes.io/projected/03b01c81-164a-46a8-8a40-da1c7131c10e-kube-api-access-c87nm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:26.246863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.246768    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"
Oct 30 13:59:26.246863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.246796    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"
Oct 30 13:59:26.246863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.246796    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98" event={"ID":"08bebcb8-9b91-4ee8-ae68-b6457e01f397","Type":"ContainerDied","Data":"e1d365a6b06d6cdd1830b2d6185a5251acc4351c030fa059bbd54c287b562f8b"}
Oct 30 13:59:26.246863 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.246865    4291 scope.go:122] "RemoveContainer" containerID="e08fe47be7e0049f88b315f0cd9e0fb31f1d39371c11a42ab07cc24f2220255b"
Oct 30 13:59:26.299161 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.299115    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"]
Oct 30 13:59:26.311049 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.311010    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-996b8b-7f8cb86bd6-q2989"]
Oct 30 13:59:26.339399 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.339356    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"]
Oct 30 13:59:26.343361 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:26.343328    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-996b8b-86b8b647-dhv98"]
Oct 30 13:59:27.882253 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:27.882210    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03b01c81-164a-46a8-8a40-da1c7131c10e" path="/var/lib/kubelet/pods/03b01c81-164a-46a8-8a40-da1c7131c10e/volumes"
Oct 30 13:59:27.882834 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:27.882816    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" path="/var/lib/kubelet/pods/08bebcb8-9b91-4ee8-ae68-b6457e01f397/volumes"
Oct 30 13:59:35.785819 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.785782    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"]
Oct 30 13:59:35.786488 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786474    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1dc6b5a3-1dd6-487e-9bba-f63c844adf06" containerName="revoke-keys"
Oct 30 13:59:35.786535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786489    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1dc6b5a3-1dd6-487e-9bba-f63c844adf06" containerName="revoke-keys"
Oct 30 13:59:35.786535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786511    4291 cpu_manager.go:414] "containerMap: removing container" podUID="03b01c81-164a-46a8-8a40-da1c7131c10e" containerName="payload-processing"
Oct 30 13:59:35.786535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786516    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03b01c81-164a-46a8-8a40-da1c7131c10e" containerName="payload-processing"
Oct 30 13:59:35.786535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786523    4291 cpu_manager.go:414] "containerMap: removing container" podUID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" containerName="payload-pre-processing"
Oct 30 13:59:35.786535 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786529    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" containerName="payload-pre-processing"
Oct 30 13:59:35.786806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786551    4291 cpu_manager.go:414] "containerMap: removing container" podUID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" containerName="payload-pre-processing"
Oct 30 13:59:35.786806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786558    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" containerName="payload-pre-processing"
Oct 30 13:59:35.786806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786663    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="08bebcb8-9b91-4ee8-ae68-b6457e01f397" containerName="payload-pre-processing"
Oct 30 13:59:35.786806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786678    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1dc6b5a3-1dd6-487e-9bba-f63c844adf06" containerName="revoke-keys"
Oct 30 13:59:35.786806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786687    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="03b01c81-164a-46a8-8a40-da1c7131c10e" containerName="payload-processing"
Oct 30 13:59:35.786806 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.786692    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f9a857-ba8b-4c06-bd76-faa29e505dbf" containerName="payload-pre-processing"
Oct 30 13:59:35.793204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.793184    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.808385 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.808355    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"]
Oct 30 13:59:35.815145 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.815113    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:59:35.815348 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.815332    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:59:35.815439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.815398    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:59:35.819324 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.819178    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:59:35.862628 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.862590    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.862838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.862739    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5l52\" (UniqueName: \"kubernetes.io/projected/92b697ce-2854-42ce-95c2-cd1993de2adf-kube-api-access-l5l52\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.862838 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.862824    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.964540 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.964500    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.964780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.964606    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5l52\" (UniqueName: \"kubernetes.io/projected/92b697ce-2854-42ce-95c2-cd1993de2adf-kube-api-access-l5l52\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.964780 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.964655    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.965249 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.965219    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.967047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.967020    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:35.984545 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:35.984517    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5l52\" (UniqueName: \"kubernetes.io/projected/92b697ce-2854-42ce-95c2-cd1993de2adf-kube-api-access-l5l52\") pod \"maas-api-revoke-sub-f18ed421-zqx78\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:36.110723 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:36.110685    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:36.425204 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:36.425167    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"]
Oct 30 13:59:36.426075 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:59:36.426046    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92b697ce_2854_42ce_95c2_cd1993de2adf.slice/crio-90e004b165aaf4bf46c9720b8b6987074b575a492932b885e6cc3ae8ce7bceb2 WatchSource:0}: Error finding container 90e004b165aaf4bf46c9720b8b6987074b575a492932b885e6cc3ae8ce7bceb2: Status 404 returned error can't find the container with id 90e004b165aaf4bf46c9720b8b6987074b575a492932b885e6cc3ae8ce7bceb2
Oct 30 13:59:37.344982 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:37.344955    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78" event={"ID":"92b697ce-2854-42ce-95c2-cd1993de2adf","Type":"ContainerStarted","Data":"951849d9c41948f9ceabe8e924a6fd61803ecc4cce2c65ba9468383a84a19443"}
Oct 30 13:59:37.345474 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:37.345007    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78" event={"ID":"92b697ce-2854-42ce-95c2-cd1993de2adf","Type":"ContainerStarted","Data":"90e004b165aaf4bf46c9720b8b6987074b575a492932b885e6cc3ae8ce7bceb2"}
Oct 30 13:59:37.387981 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:37.387931    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78" podStartSLOduration=2.387916385 podStartE2EDuration="2.387916385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:37.383365738 +0000 UTC m=+3387.643603972" watchObservedRunningTime="2026-10-30 13:59:37.387916385 +0000 UTC m=+3387.648154619"
Oct 30 13:59:38.220321 ip-10-0-136-36 kubenswrapper[4291]: E1030 13:59:38.220274    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 13:59:38.355197 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:38.355157    4291 generic.go:356] "Generic (PLEG): container finished" podID="92b697ce-2854-42ce-95c2-cd1993de2adf" containerID="951849d9c41948f9ceabe8e924a6fd61803ecc4cce2c65ba9468383a84a19443" exitCode=0
Oct 30 13:59:38.355699 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:38.355243    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78" event={"ID":"92b697ce-2854-42ce-95c2-cd1993de2adf","Type":"ContainerDied","Data":"951849d9c41948f9ceabe8e924a6fd61803ecc4cce2c65ba9468383a84a19443"}
Oct 30 13:59:39.534575 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.534551    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:39.616862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.616772    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-xks-ca\") pod \"92b697ce-2854-42ce-95c2-cd1993de2adf\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") "
Oct 30 13:59:39.616862 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.616825    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-openshift-ca\") pod \"92b697ce-2854-42ce-95c2-cd1993de2adf\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") "
Oct 30 13:59:39.617071 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.617048    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92b697ce-2854-42ce-95c2-cd1993de2adf-kube-api-access-l5l52\" (UniqueName: \"kubernetes.io/projected/92b697ce-2854-42ce-95c2-cd1993de2adf-kube-api-access-l5l52\") pod \"92b697ce-2854-42ce-95c2-cd1993de2adf\" (UID: \"92b697ce-2854-42ce-95c2-cd1993de2adf\") "
Oct 30 13:59:39.617287 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.617259    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-openshift-ca" pod "92b697ce-2854-42ce-95c2-cd1993de2adf" (UID: "92b697ce-2854-42ce-95c2-cd1993de2adf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:39.617664 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.617612    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:39.619047 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.619019    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-xks-ca" pod "92b697ce-2854-42ce-95c2-cd1993de2adf" (UID: "92b697ce-2854-42ce-95c2-cd1993de2adf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:59:39.619188 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.619168    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92b697ce-2854-42ce-95c2-cd1993de2adf-kube-api-access-l5l52" pod "92b697ce-2854-42ce-95c2-cd1993de2adf" (UID: "92b697ce-2854-42ce-95c2-cd1993de2adf"). InnerVolumeSpecName "kube-api-access-l5l52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:39.719007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.718969    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5l52\" (UniqueName: \"kubernetes.io/projected/92b697ce-2854-42ce-95c2-cd1993de2adf-kube-api-access-l5l52\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:39.719007 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:39.718998    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/92b697ce-2854-42ce-95c2-cd1993de2adf-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:40.381910 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:40.381867    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78" event={"ID":"92b697ce-2854-42ce-95c2-cd1993de2adf","Type":"ContainerDied","Data":"90e004b165aaf4bf46c9720b8b6987074b575a492932b885e6cc3ae8ce7bceb2"}
Oct 30 13:59:40.382129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:40.381961    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90e004b165aaf4bf46c9720b8b6987074b575a492932b885e6cc3ae8ce7bceb2"
Oct 30 13:59:40.382129 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:40.381878    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"
Oct 30 13:59:47.388997 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.388959    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"]
Oct 30 13:59:47.389757 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.389738    4291 cpu_manager.go:414] "containerMap: removing container" podUID="92b697ce-2854-42ce-95c2-cd1993de2adf" containerName="revoke-keys"
Oct 30 13:59:47.389818 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.389757    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92b697ce-2854-42ce-95c2-cd1993de2adf" containerName="revoke-keys"
Oct 30 13:59:47.389893 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.389884    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="92b697ce-2854-42ce-95c2-cd1993de2adf" containerName="revoke-keys"
Oct 30 13:59:47.393498 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.393482    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.416786 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.416747    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"]
Oct 30 13:59:47.422378 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.422352    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 13:59:47.422580 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.422509    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 13:59:47.502835 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.492901    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 13:59:47.517704 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.503224    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 13:59:47.584365 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.584329    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8dc6\" (UniqueName: \"kubernetes.io/projected/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-kube-api-access-k8dc6\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.584564 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.584526    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.584721 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.584702    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.686601 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.686507    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.686779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.686652    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.686779 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.686765    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8dc6\" (UniqueName: \"kubernetes.io/projected/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-kube-api-access-k8dc6\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.687360 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.687333    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.689069 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.689044    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:47.708075 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:47.708050    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8dc6\" (UniqueName: \"kubernetes.io/projected/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-kube-api-access-k8dc6\") pod \"maas-api-revoke-sub-1019287e-82227\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:48.008179 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:48.008087    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:48.315439 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:48.315409    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"]
Oct 30 13:59:48.316347 ip-10-0-136-36 kubenswrapper[4291]: W1030 13:59:48.316321    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88b8479a_41f2_40ad_9cbc_9300dc7bcd9c.slice/crio-4caf7f3782910eeddafd75055754f3ceb52bde497f32ed956ff3bb2ebce4cd6a WatchSource:0}: Error finding container 4caf7f3782910eeddafd75055754f3ceb52bde497f32ed956ff3bb2ebce4cd6a: Status 404 returned error can't find the container with id 4caf7f3782910eeddafd75055754f3ceb52bde497f32ed956ff3bb2ebce4cd6a
Oct 30 13:59:48.447012 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:48.446978    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227" event={"ID":"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c","Type":"ContainerStarted","Data":"4caf7f3782910eeddafd75055754f3ceb52bde497f32ed956ff3bb2ebce4cd6a"}
Oct 30 13:59:49.461392 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:49.461348    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227" event={"ID":"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c","Type":"ContainerStarted","Data":"239a4eb03ab693a0a83be255bfa377a14ff8931ebf4f72a74aa4f0358cf9747f"}
Oct 30 13:59:49.505758 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:49.505700    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227" podStartSLOduration=2.505684618 podStartE2EDuration="2.505684618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 13:59:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 13:59:49.502188434 +0000 UTC m=+3399.762426668" watchObservedRunningTime="2026-10-30 13:59:49.505684618 +0000 UTC m=+3399.765922851"
Oct 30 13:59:49.620794 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:49.620755    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"]
Oct 30 13:59:49.624681 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:49.624651    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-996b8b-snttz"]
Oct 30 13:59:49.881268 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:49.881236    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4da5e508-19a5-4bfe-94e2-88f8e689e3ba" path="/var/lib/kubelet/pods/4da5e508-19a5-4bfe-94e2-88f8e689e3ba/volumes"
Oct 30 13:59:50.473136 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:50.473103    4291 generic.go:356] "Generic (PLEG): container finished" podID="88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" containerID="239a4eb03ab693a0a83be255bfa377a14ff8931ebf4f72a74aa4f0358cf9747f" exitCode=0
Oct 30 13:59:50.473541 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:50.473186    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227" event={"ID":"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c","Type":"ContainerDied","Data":"239a4eb03ab693a0a83be255bfa377a14ff8931ebf4f72a74aa4f0358cf9747f"}
Oct 30 13:59:51.695781 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.695757    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:51.761248 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.761167    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-xks-ca\") pod \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") "
Oct 30 13:59:51.761407 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.761329    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-kube-api-access-k8dc6\" (UniqueName: \"kubernetes.io/projected/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-kube-api-access-k8dc6\") pod \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") "
Oct 30 13:59:51.761486 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.761472    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-openshift-ca\") pod \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\" (UID: \"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c\") "
Oct 30 13:59:51.762275 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.762243    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-openshift-ca" pod "88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" (UID: "88b8479a-41f2-40ad-9cbc-9300dc7bcd9c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 13:59:51.763733 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.763703    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-xks-ca" pod "88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" (UID: "88b8479a-41f2-40ad-9cbc-9300dc7bcd9c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 13:59:51.764362 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.764335    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-kube-api-access-k8dc6" pod "88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" (UID: "88b8479a-41f2-40ad-9cbc-9300dc7bcd9c"). InnerVolumeSpecName "kube-api-access-k8dc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 13:59:51.863098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.863065    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8dc6\" (UniqueName: \"kubernetes.io/projected/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-kube-api-access-k8dc6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:51.863098 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.863098    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:51.863258 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:51.863116    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 13:59:52.489421 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:52.489392    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"
Oct 30 13:59:52.489600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:52.489427    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227" event={"ID":"88b8479a-41f2-40ad-9cbc-9300dc7bcd9c","Type":"ContainerDied","Data":"4caf7f3782910eeddafd75055754f3ceb52bde497f32ed956ff3bb2ebce4cd6a"}
Oct 30 13:59:52.489600 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:52.489478    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4caf7f3782910eeddafd75055754f3ceb52bde497f32ed956ff3bb2ebce4cd6a"
Oct 30 13:59:56.285457 ip-10-0-136-36 kubenswrapper[4291]: I1030 13:59:56.285425    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:00:00.152936 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.152897    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"]
Oct 30 14:00:00.153629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.153611    4291 cpu_manager.go:414] "containerMap: removing container" podUID="88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" containerName="revoke-keys"
Oct 30 14:00:00.153700 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.153631    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" containerName="revoke-keys"
Oct 30 14:00:00.153796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.153786    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" containerName="revoke-keys"
Oct 30 14:00:00.157785 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.157762    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"]
Oct 30 14:00:00.157938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.157878    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.162178 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.162157    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"]
Oct 30 14:00:00.162317 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.162296    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.166370 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.166351    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"]
Oct 30 14:00:00.166491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.166450    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.170759 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.170742    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"]
Oct 30 14:00:00.170923 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.170864    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.174927 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.174907    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.180108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.180081    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"]
Oct 30 14:00:00.180413 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.180392    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:00:00.180571 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.180502    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:00:00.180687 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.180617    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:00:00.187279 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.187064    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"]
Oct 30 14:00:00.205028 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.204986    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/addb6a6e-6673-4881-b84c-fa9561235a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.238812 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.238776    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47367ecc-1223-4952-a0b8-986afe4951b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.239166 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.239141    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndh64\" (UniqueName: \"kubernetes.io/projected/421d7f0d-0d50-4985-95f4-4058c691b840-kube-api-access-ndh64\") pod \"maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.239332 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.239313    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421d7f0d-0d50-4985-95f4-4058c691b840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.239521 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.239505    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngl9z\" (UniqueName: \"kubernetes.io/projected/47367ecc-1223-4952-a0b8-986afe4951b9-kube-api-access-ngl9z\") pod \"maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.240057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.240034    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76cwp\" (UniqueName: \"kubernetes.io/projected/addb6a6e-6673-4881-b84c-fa9561235a47-kube-api-access-76cwp\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.251820 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.251780    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"]
Oct 30 14:00:00.256718 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.256692    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"]
Oct 30 14:00:00.267656 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.264560    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"]
Oct 30 14:00:00.308725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.307230    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"]
Oct 30 14:00:00.313343 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.312038    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.341868    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngl9z\" (UniqueName: \"kubernetes.io/projected/47367ecc-1223-4952-a0b8-986afe4951b9-kube-api-access-ngl9z\") pod \"maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.341954    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76cwp\" (UniqueName: \"kubernetes.io/projected/addb6a6e-6673-4881-b84c-fa9561235a47-kube-api-access-76cwp\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342039    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5fc286-5675-4c60-bb29-9b0bf79bd701-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342091    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg262\" (UniqueName: \"kubernetes.io/projected/5b5fc286-5675-4c60-bb29-9b0bf79bd701-kube-api-access-jg262\") pod \"maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342174    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct7z9\" (UniqueName: \"kubernetes.io/projected/d0e7264f-66cf-45f6-91db-4126497c2ae6-kube-api-access-ct7z9\") pod \"maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342604    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/addb6a6e-6673-4881-b84c-fa9561235a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342674    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47367ecc-1223-4952-a0b8-986afe4951b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342703    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7264f-66cf-45f6-91db-4126497c2ae6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342751    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndh64\" (UniqueName: \"kubernetes.io/projected/421d7f0d-0d50-4985-95f4-4058c691b840-kube-api-access-ndh64\") pod \"maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.343012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.342787    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421d7f0d-0d50-4985-95f4-4058c691b840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.343745 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.343591    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421d7f0d-0d50-4985-95f4-4058c691b840-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.345286 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.344989    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/addb6a6e-6673-4881-b84c-fa9561235a47-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.345286 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.345266    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47367ecc-1223-4952-a0b8-986afe4951b9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.345951 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.345931    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"]
Oct 30 14:00:00.346164 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.346054    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.358676 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.354813    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"]
Oct 30 14:00:00.359454 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.359437    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.360330 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.360312    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"]
Oct 30 14:00:00.360685 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.360665    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.369788 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.369759    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"]
Oct 30 14:00:00.369933 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.369810    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"]
Oct 30 14:00:00.370007 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.369982    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.371477 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.371443    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76cwp\" (UniqueName: \"kubernetes.io/projected/addb6a6e-6673-4881-b84c-fa9561235a47-kube-api-access-76cwp\") pod \"maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.381091 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.381062    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"]
Oct 30 14:00:00.383309 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.383288    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.387386 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.387144    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngl9z\" (UniqueName: \"kubernetes.io/projected/47367ecc-1223-4952-a0b8-986afe4951b9-kube-api-access-ngl9z\") pod \"maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.387506 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.387296    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"]
Oct 30 14:00:00.387506 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.387443    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"]
Oct 30 14:00:00.387506 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.387470    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"]
Oct 30 14:00:00.387506 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.387497    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"]
Oct 30 14:00:00.390752 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.387359    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.398835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.394375    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"]
Oct 30 14:00:00.398835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.394472    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.398835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.397582    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndh64\" (UniqueName: \"kubernetes.io/projected/421d7f0d-0d50-4985-95f4-4058c691b840-kube-api-access-ndh64\") pod \"maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.401872 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.401827    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"]
Oct 30 14:00:00.402028 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.402011    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.421222 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.421139    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"]
Oct 30 14:00:00.422359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.422330    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.445558 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.445519    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmswf\" (UniqueName: \"kubernetes.io/projected/398137cb-99db-4931-bca0-f6e1b1b931a7-kube-api-access-rmswf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.445754 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.445654    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljw5j\" (UniqueName: \"kubernetes.io/projected/5d440f70-f544-432c-a8f1-e38d9cb7ec68-kube-api-access-ljw5j\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.445872 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.445848    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7264f-66cf-45f6-91db-4126497c2ae6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.445947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.445905    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f764f2cc-15c4-460c-930c-9b16c9fa82ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.446155 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.445993    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6941b-9f8e-4ca3-84cc-7fa377906a40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.446699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.446651    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5l6g\" (UniqueName: \"kubernetes.io/projected/f764f2cc-15c4-460c-930c-9b16c9fa82ab-kube-api-access-p5l6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.446819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.446709    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvv5v\" (UniqueName: \"kubernetes.io/projected/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-kube-api-access-bvv5v\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.446819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.446810    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/398137cb-99db-4931-bca0-f6e1b1b931a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.446952 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.446925    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5fc286-5675-4c60-bb29-9b0bf79bd701-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.447014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.446981    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d440f70-f544-432c-a8f1-e38d9cb7ec68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.447546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.447024    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg262\" (UniqueName: \"kubernetes.io/projected/5b5fc286-5675-4c60-bb29-9b0bf79bd701-kube-api-access-jg262\") pod \"maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.447546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.447086    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljsjc\" (UniqueName: \"kubernetes.io/projected/02c6941b-9f8e-4ca3-84cc-7fa377906a40-kube-api-access-ljsjc\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.447546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.447124    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.447546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.447237    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct7z9\" (UniqueName: \"kubernetes.io/projected/d0e7264f-66cf-45f6-91db-4126497c2ae6-kube-api-access-ct7z9\") pod \"maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.447546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.447362    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7264f-66cf-45f6-91db-4126497c2ae6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.448530 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.448504    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5fc286-5675-4c60-bb29-9b0bf79bd701-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.487686 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.477004    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:00.487686 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.480787    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"]
Oct 30 14:00:00.487686 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.481235    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:00.492513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.491628    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct7z9\" (UniqueName: \"kubernetes.io/projected/d0e7264f-66cf-45f6-91db-4126497c2ae6-kube-api-access-ct7z9\") pod \"maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.493870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.492848    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"]
Oct 30 14:00:00.493870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.492892    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"]
Oct 30 14:00:00.493870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.492920    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"]
Oct 30 14:00:00.493870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.492945    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"]
Oct 30 14:00:00.493870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.492984    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.500585 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.499142    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"]
Oct 30 14:00:00.500585 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.499767    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg262\" (UniqueName: \"kubernetes.io/projected/5b5fc286-5675-4c60-bb29-9b0bf79bd701-kube-api-access-jg262\") pod \"maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.505921 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.505898    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"]
Oct 30 14:00:00.506031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.506007    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:00.509154 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.509128    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"]
Oct 30 14:00:00.516928 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.516906    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:00.524970 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.524951    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"]
Oct 30 14:00:00.532070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.532046    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"]
Oct 30 14:00:00.539538 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.539507    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"]
Oct 30 14:00:00.544368 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.544339    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"]
Oct 30 14:00:00.544695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.544424    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:00.549088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549052    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5l6g\" (UniqueName: \"kubernetes.io/projected/f764f2cc-15c4-460c-930c-9b16c9fa82ab-kube-api-access-p5l6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.549260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549097    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvv5v\" (UniqueName: \"kubernetes.io/projected/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-kube-api-access-bvv5v\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.549260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549123    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/cc3d3284-1405-4e29-91fd-fba47df039df-kube-api-access-2t4vz\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:00.549260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549160    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b7f3d-c901-4f59-a039-04c48e60d984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.549260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549189    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851ad965-5bcf-46ed-bdf4-1764a61b901a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.549260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549237    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/398137cb-99db-4931-bca0-f6e1b1b931a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549270    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4ks7\" (UniqueName: \"kubernetes.io/projected/851ad965-5bcf-46ed-bdf4-1764a61b901a-kube-api-access-d4ks7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549340    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cgvk\" (UniqueName: \"kubernetes.io/projected/e43b7f3d-c901-4f59-a039-04c48e60d984-kube-api-access-8cgvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549362    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d440f70-f544-432c-a8f1-e38d9cb7ec68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549384    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nrr2\" (UniqueName: \"kubernetes.io/projected/077541bb-8a79-4080-8a2c-2644004dbe3d-kube-api-access-6nrr2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549427    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljsjc\" (UniqueName: \"kubernetes.io/projected/02c6941b-9f8e-4ca3-84cc-7fa377906a40-kube-api-access-ljsjc\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549444    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549468    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5l8z\" (UniqueName: \"kubernetes.io/projected/40ba6924-2d7d-41bc-9749-458e7a705e64-kube-api-access-s5l8z\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.549545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549496    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f297ee2-5036-4ac9-8839-c80d2eb293e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549567    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d3284-1405-4e29-91fd-fba47df039df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549603    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmswf\" (UniqueName: \"kubernetes.io/projected/398137cb-99db-4931-bca0-f6e1b1b931a7-kube-api-access-rmswf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549698    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/077541bb-8a79-4080-8a2c-2644004dbe3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549730    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljw5j\" (UniqueName: \"kubernetes.io/projected/5d440f70-f544-432c-a8f1-e38d9cb7ec68-kube-api-access-ljw5j\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549761    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xxf5\" (UniqueName: \"kubernetes.io/projected/8f297ee2-5036-4ac9-8839-c80d2eb293e5-kube-api-access-4xxf5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549792    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ba6924-2d7d-41bc-9749-458e7a705e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549860    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f764f2cc-15c4-460c-930c-9b16c9fa82ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.550031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.549882    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6941b-9f8e-4ca3-84cc-7fa377906a40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.550564 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.550533    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6941b-9f8e-4ca3-84cc-7fa377906a40-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.551566 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.551535    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/398137cb-99db-4931-bca0-f6e1b1b931a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.552197 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.552174    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d440f70-f544-432c-a8f1-e38d9cb7ec68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.556011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.553493    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"]
Oct 30 14:00:00.556011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.553551    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f764f2cc-15c4-460c-930c-9b16c9fa82ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.556011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.553777    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:00.556011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.554092    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.558159 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.558121    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"]
Oct 30 14:00:00.564857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.564828    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"]
Oct 30 14:00:00.574655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.572969    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"]
Oct 30 14:00:00.574655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.573087    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:00.576040 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.575984    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljw5j\" (UniqueName: \"kubernetes.io/projected/5d440f70-f544-432c-a8f1-e38d9cb7ec68-kube-api-access-ljw5j\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.577206 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.577170    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5l6g\" (UniqueName: \"kubernetes.io/projected/f764f2cc-15c4-460c-930c-9b16c9fa82ab-kube-api-access-p5l6g\") pod \"maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.578801 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.578366    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmswf\" (UniqueName: \"kubernetes.io/projected/398137cb-99db-4931-bca0-f6e1b1b931a7-kube-api-access-rmswf\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.584443 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.584419    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:00.585337 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.585273    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvv5v\" (UniqueName: \"kubernetes.io/projected/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-kube-api-access-bvv5v\") pod \"maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.594291 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.587429    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljsjc\" (UniqueName: \"kubernetes.io/projected/02c6941b-9f8e-4ca3-84cc-7fa377906a40-kube-api-access-ljsjc\") pod \"maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.594291 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.588842    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"]
Oct 30 14:00:00.594291 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.593935    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:00.612823 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.605885    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"]
Oct 30 14:00:00.612823 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.610566    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"]
Oct 30 14:00:00.617370 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.617342    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"]
Oct 30 14:00:00.621682 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.620915    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:00.651242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.651214    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f297ee2-5036-4ac9-8839-c80d2eb293e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.651450 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.651425    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d3284-1405-4e29-91fd-fba47df039df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:00.651551 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.651533    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/077541bb-8a79-4080-8a2c-2644004dbe3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.651689 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.651665    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xxf5\" (UniqueName: \"kubernetes.io/projected/8f297ee2-5036-4ac9-8839-c80d2eb293e5-kube-api-access-4xxf5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.651772 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.651746    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ba6924-2d7d-41bc-9749-458e7a705e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.651852 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.651791    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:00.651920 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.651852    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pzmg\" (UniqueName: \"kubernetes.io/projected/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-kube-api-access-2pzmg\") pod \"maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:00.652199 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652171    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a630f92-dc84-454b-8f37-c80cdcfe40b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889480-hkr66\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652286    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f297ee2-5036-4ac9-8839-c80d2eb293e5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652313    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/cc3d3284-1405-4e29-91fd-fba47df039df-kube-api-access-2t4vz\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652412    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/077541bb-8a79-4080-8a2c-2644004dbe3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652353    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2804875f-4c19-409b-8646-c17b8507f6f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652604    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkvhz\" (UniqueName: \"kubernetes.io/projected/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-kube-api-access-kkvhz\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652680    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b7f3d-c901-4f59-a039-04c48e60d984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652741    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851ad965-5bcf-46ed-bdf4-1764a61b901a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652792    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqlzs\" (UniqueName: \"kubernetes.io/projected/2804875f-4c19-409b-8646-c17b8507f6f2-kube-api-access-tqlzs\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652838    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652920    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4ks7\" (UniqueName: \"kubernetes.io/projected/851ad965-5bcf-46ed-bdf4-1764a61b901a-kube-api-access-d4ks7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.652983    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.653035    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d3284-1405-4e29-91fd-fba47df039df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:00.653327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.653052    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cgvk\" (UniqueName: \"kubernetes.io/projected/e43b7f3d-c901-4f59-a039-04c48e60d984-kube-api-access-8cgvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.654263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.653983    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nrr2\" (UniqueName: \"kubernetes.io/projected/077541bb-8a79-4080-8a2c-2644004dbe3d-kube-api-access-6nrr2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.654263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.653980    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851ad965-5bcf-46ed-bdf4-1764a61b901a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.655009 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.654552    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqdkk\" (UniqueName: \"kubernetes.io/projected/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-kube-api-access-gqdkk\") pod \"maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:00.655009 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.654683    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4b5h\" (UniqueName: \"kubernetes.io/projected/9a630f92-dc84-454b-8f37-c80cdcfe40b1-kube-api-access-d4b5h\") pod \"maas-api-key-cleanup-29889480-hkr66\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:00.655009 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.654721    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5l8z\" (UniqueName: \"kubernetes.io/projected/40ba6924-2d7d-41bc-9749-458e7a705e64-kube-api-access-s5l8z\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.655009 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.654835    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b7f3d-c901-4f59-a039-04c48e60d984-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.655270 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.655030    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ba6924-2d7d-41bc-9749-458e7a705e64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.661155 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.659817    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:00.678178 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.678060    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4ks7\" (UniqueName: \"kubernetes.io/projected/851ad965-5bcf-46ed-bdf4-1764a61b901a-kube-api-access-d4ks7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.680124 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.680096    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5l8z\" (UniqueName: \"kubernetes.io/projected/40ba6924-2d7d-41bc-9749-458e7a705e64-kube-api-access-s5l8z\") pod \"maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.684776 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.684747    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xxf5\" (UniqueName: \"kubernetes.io/projected/8f297ee2-5036-4ac9-8839-c80d2eb293e5-kube-api-access-4xxf5\") pod \"maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.686079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.686052    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:00.691093 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.690086    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nrr2\" (UniqueName: \"kubernetes.io/projected/077541bb-8a79-4080-8a2c-2644004dbe3d-kube-api-access-6nrr2\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.691093 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.691046    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cgvk\" (UniqueName: \"kubernetes.io/projected/e43b7f3d-c901-4f59-a039-04c48e60d984-kube-api-access-8cgvk\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.703565 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.703501    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/cc3d3284-1405-4e29-91fd-fba47df039df-kube-api-access-2t4vz\") pod \"maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:00.752200 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.751930    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.755758    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqdkk\" (UniqueName: \"kubernetes.io/projected/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-kube-api-access-gqdkk\") pod \"maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.755823    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4b5h\" (UniqueName: \"kubernetes.io/projected/9a630f92-dc84-454b-8f37-c80cdcfe40b1-kube-api-access-d4b5h\") pod \"maas-api-key-cleanup-29889480-hkr66\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.755917    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756061    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756085    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-272tx\" (UniqueName: \"kubernetes.io/projected/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-kube-api-access-272tx\") pod \"maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756118    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pzmg\" (UniqueName: \"kubernetes.io/projected/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-kube-api-access-2pzmg\") pod \"maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756168    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a630f92-dc84-454b-8f37-c80cdcfe40b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889480-hkr66\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756234    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2804875f-4c19-409b-8646-c17b8507f6f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756267    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkvhz\" (UniqueName: \"kubernetes.io/projected/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-kube-api-access-kkvhz\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756333    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqlzs\" (UniqueName: \"kubernetes.io/projected/2804875f-4c19-409b-8646-c17b8507f6f2-kube-api-access-tqlzs\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756376    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.756471    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.757398    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.759483    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.760658    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a630f92-dc84-454b-8f37-c80cdcfe40b1-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889480-hkr66\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:00.762784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.761388    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2804875f-4c19-409b-8646-c17b8507f6f2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.765697 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.765234    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:00.781359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.781325    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkvhz\" (UniqueName: \"kubernetes.io/projected/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-kube-api-access-kkvhz\") pod \"maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:00.781479 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.781360    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pzmg\" (UniqueName: \"kubernetes.io/projected/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-kube-api-access-2pzmg\") pod \"maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:00.781551 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.781537    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:00.782083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.781930    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqdkk\" (UniqueName: \"kubernetes.io/projected/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-kube-api-access-gqdkk\") pod \"maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:00.783193 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.783101    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4b5h\" (UniqueName: \"kubernetes.io/projected/9a630f92-dc84-454b-8f37-c80cdcfe40b1-kube-api-access-d4b5h\") pod \"maas-api-key-cleanup-29889480-hkr66\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:00.784555 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.784508    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqlzs\" (UniqueName: \"kubernetes.io/projected/2804875f-4c19-409b-8646-c17b8507f6f2-kube-api-access-tqlzs\") pod \"maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.835978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.835942    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:00.846764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.844784    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:00.854105 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.854029    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"]
Oct 30 14:00:00.862483 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.858029    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:00.862483 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.858137    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-272tx\" (UniqueName: \"kubernetes.io/projected/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-kube-api-access-272tx\") pod \"maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:00.862483 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.861724    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:00.865460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.865253    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:00.876158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.876132    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:00.919055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.919024    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-272tx\" (UniqueName: \"kubernetes.io/projected/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-kube-api-access-272tx\") pod \"maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:00.929626 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.929563    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:00.937140 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.937107    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:00.950387 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.950347    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:00.987675 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.987460    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:00.988100 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:00.988047    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:01.006235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.006205    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:01.029135 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.028894    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:01.048755 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.048396    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:01.072455 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.072266    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:01.080121 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.079903    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:01.121048 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.121023    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"]
Oct 30 14:00:01.144679 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.144621    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaddb6a6e_6673_4881_b84c_fa9561235a47.slice/crio-042d0f693d1b060f5f40fc6a57a11a0399f65c268f124d3bed24e285c71c5929 WatchSource:0}: Error finding container 042d0f693d1b060f5f40fc6a57a11a0399f65c268f124d3bed24e285c71c5929: Status 404 returned error can't find the container with id 042d0f693d1b060f5f40fc6a57a11a0399f65c268f124d3bed24e285c71c5929
Oct 30 14:00:01.487802 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.487751    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"]
Oct 30 14:00:01.489697 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.489607    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf764f2cc_15c4_460c_930c_9b16c9fa82ab.slice/crio-b00f4d7cc72945bfa4712e855fd65553701928654c1982b3caf82ba027ef8126 WatchSource:0}: Error finding container b00f4d7cc72945bfa4712e855fd65553701928654c1982b3caf82ba027ef8126: Status 404 returned error can't find the container with id b00f4d7cc72945bfa4712e855fd65553701928654c1982b3caf82ba027ef8126
Oct 30 14:00:01.523902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.523876    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"]
Oct 30 14:00:01.544701 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.544345    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf93e80a_eb0c_447c_a9c6_a37be5c77ab0.slice/crio-fc71fbd89d6e9d52e4fba30b0cb43736517c2a0c1570b77d78d896cec5d93e68 WatchSource:0}: Error finding container fc71fbd89d6e9d52e4fba30b0cb43736517c2a0c1570b77d78d896cec5d93e68: Status 404 returned error can't find the container with id fc71fbd89d6e9d52e4fba30b0cb43736517c2a0c1570b77d78d896cec5d93e68
Oct 30 14:00:01.573336 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.573141    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"]
Oct 30 14:00:01.585083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.584033    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"]
Oct 30 14:00:01.617617 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.617586    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"]
Oct 30 14:00:01.633572 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.632471    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerStarted","Data":"dcd982b060f364dc84d5ebf1dc105ad5b28b3c094d84d8d720b5051734c29680"}
Oct 30 14:00:01.633572 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.632546    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerStarted","Data":"df678e237379d173f3475c712ca8c757d6b84a366be6cd1aed3ded1806f7e287"}
Oct 30 14:00:01.635560 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.635483    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" event={"ID":"addb6a6e-6673-4881-b84c-fa9561235a47","Type":"ContainerStarted","Data":"042d0f693d1b060f5f40fc6a57a11a0399f65c268f124d3bed24e285c71c5929"}
Oct 30 14:00:01.643699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.643456    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"]
Oct 30 14:00:01.649154 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.649125    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" event={"ID":"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0","Type":"ContainerStarted","Data":"fc71fbd89d6e9d52e4fba30b0cb43736517c2a0c1570b77d78d896cec5d93e68"}
Oct 30 14:00:01.650352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.650331    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"]
Oct 30 14:00:01.654176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.654153    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" event={"ID":"f764f2cc-15c4-460c-930c-9b16c9fa82ab","Type":"ContainerStarted","Data":"b00f4d7cc72945bfa4712e855fd65553701928654c1982b3caf82ba027ef8126"}
Oct 30 14:00:01.657101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.657075    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"]
Oct 30 14:00:01.658263 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.658237    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c6941b_9f8e_4ca3_84cc_7fa377906a40.slice/crio-c32b5a43612b8e0d2ec988e03407f50d02c2969c1eef434ce1dcfb2af7be3718 WatchSource:0}: Error finding container c32b5a43612b8e0d2ec988e03407f50d02c2969c1eef434ce1dcfb2af7be3718: Status 404 returned error can't find the container with id c32b5a43612b8e0d2ec988e03407f50d02c2969c1eef434ce1dcfb2af7be3718
Oct 30 14:00:01.658889 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.658845    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b5fc286_5675_4c60_bb29_9b0bf79bd701.slice/crio-94f357dcfe58dcacca772fffbbf523421ad8db8d56d96c01289757d2ff47c1e6 WatchSource:0}: Error finding container 94f357dcfe58dcacca772fffbbf523421ad8db8d56d96c01289757d2ff47c1e6: Status 404 returned error can't find the container with id 94f357dcfe58dcacca772fffbbf523421ad8db8d56d96c01289757d2ff47c1e6
Oct 30 14:00:01.660187 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.660161    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43b7f3d_c901_4f59_a039_04c48e60d984.slice/crio-90ee5372eb01592a399bdf52e38b0105f0f1120dba143bca152f0f454902c21c WatchSource:0}: Error finding container 90ee5372eb01592a399bdf52e38b0105f0f1120dba143bca152f0f454902c21c: Status 404 returned error can't find the container with id 90ee5372eb01592a399bdf52e38b0105f0f1120dba143bca152f0f454902c21c
Oct 30 14:00:01.673717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.673653    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" podStartSLOduration=1.6736228579999999 podStartE2EDuration="1.673622858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:01.671596181 +0000 UTC m=+3411.931834416" watchObservedRunningTime="2026-10-30 14:00:01.673622858 +0000 UTC m=+3411.933861092"
Oct 30 14:00:01.727951 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.727919    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"]
Oct 30 14:00:01.731156 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.731123    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d440f70_f544_432c_a8f1_e38d9cb7ec68.slice/crio-7e9a849f5d5c72687a11605a52995584d187eaf8d293101d60f1e59018bdb176 WatchSource:0}: Error finding container 7e9a849f5d5c72687a11605a52995584d187eaf8d293101d60f1e59018bdb176: Status 404 returned error can't find the container with id 7e9a849f5d5c72687a11605a52995584d187eaf8d293101d60f1e59018bdb176
Oct 30 14:00:01.796852 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.796813    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"]
Oct 30 14:00:01.803704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.803672    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"]
Oct 30 14:00:01.825541 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.825497    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"]
Oct 30 14:00:01.853628 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.853590    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"]
Oct 30 14:00:01.953607 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.953251    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40ba6924_2d7d_41bc_9749_458e7a705e64.slice/crio-a77052963a1dcd269d973ec37f3e32abe2ba0ca54801a1cdc0cf4b87fe74cc66 WatchSource:0}: Error finding container a77052963a1dcd269d973ec37f3e32abe2ba0ca54801a1cdc0cf4b87fe74cc66: Status 404 returned error can't find the container with id a77052963a1dcd269d973ec37f3e32abe2ba0ca54801a1cdc0cf4b87fe74cc66
Oct 30 14:00:01.962786 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:01.962734    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2804875f_4c19_409b_8646_c17b8507f6f2.slice/crio-dad9960baed21831b000e11b66d7137dff3f8d65910941ec87ae6d7e1613567b WatchSource:0}: Error finding container dad9960baed21831b000e11b66d7137dff3f8d65910941ec87ae6d7e1613567b: Status 404 returned error can't find the container with id dad9960baed21831b000e11b66d7137dff3f8d65910941ec87ae6d7e1613567b
Oct 30 14:00:01.999767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.999611    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"]
Oct 30 14:00:01.999767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.999694    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"]
Oct 30 14:00:01.999767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.999727    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"]
Oct 30 14:00:01.999941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.999815    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"]
Oct 30 14:00:01.999941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.999842    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"]
Oct 30 14:00:01.999941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:01.999869    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"]
Oct 30 14:00:02.008657 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.008596    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"]
Oct 30 14:00:02.674041 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.674005    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerStarted","Data":"80ffff421692a56aea710de3e5afdd430360216c5fd82253f263f87457c0c90c"}
Oct 30 14:00:02.676347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.676321    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" event={"ID":"02c6941b-9f8e-4ca3-84cc-7fa377906a40","Type":"ContainerStarted","Data":"c32b5a43612b8e0d2ec988e03407f50d02c2969c1eef434ce1dcfb2af7be3718"}
Oct 30 14:00:02.683082 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.678970    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4" event={"ID":"5b5fc286-5675-4c60-bb29-9b0bf79bd701","Type":"ContainerStarted","Data":"94f357dcfe58dcacca772fffbbf523421ad8db8d56d96c01289757d2ff47c1e6"}
Oct 30 14:00:02.683082 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.682177    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" event={"ID":"cc3d3284-1405-4e29-91fd-fba47df039df","Type":"ContainerStarted","Data":"86b84a7f03d687297b7e0c071f8426d11b3ea552d74e9c802fd325a33665da22"}
Oct 30 14:00:02.685966 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.685912    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" event={"ID":"e43b7f3d-c901-4f59-a039-04c48e60d984","Type":"ContainerStarted","Data":"90ee5372eb01592a399bdf52e38b0105f0f1120dba143bca152f0f454902c21c"}
Oct 30 14:00:02.688399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.688377    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" event={"ID":"5d440f70-f544-432c-a8f1-e38d9cb7ec68","Type":"ContainerStarted","Data":"a04255785fc734007fc6e1994fc45a5db63baff9be329299691bd0578d984c87"}
Oct 30 14:00:02.688510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.688421    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" event={"ID":"5d440f70-f544-432c-a8f1-e38d9cb7ec68","Type":"ContainerStarted","Data":"7e9a849f5d5c72687a11605a52995584d187eaf8d293101d60f1e59018bdb176"}
Oct 30 14:00:02.691534 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.691467    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" event={"ID":"addb6a6e-6673-4881-b84c-fa9561235a47","Type":"ContainerStarted","Data":"9e4adf242e78e599a7f537df4b9b9f1a89b8b60aa7b1803fda80413dc70f7eef"}
Oct 30 14:00:02.691963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.691859    4291 scope.go:122] "RemoveContainer" containerID="9e4adf242e78e599a7f537df4b9b9f1a89b8b60aa7b1803fda80413dc70f7eef"
Oct 30 14:00:02.693491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.693465    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerStarted","Data":"e030a969bb21be2fd96ca1cbd80b45b4ff65448eff2208e0bbb454374aba0b62"}
Oct 30 14:00:02.693569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.693517    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerStarted","Data":"c23477f38d15acc6d5c3bbf2257d688905c1ab304f7ce214be24774d3c0fe93d"}
Oct 30 14:00:02.695595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.695574    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" event={"ID":"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0","Type":"ContainerStarted","Data":"8ed5433b2cf5b8802c2bc76f0eab7286b4d430d7253391697aae56a750a85c5e"}
Oct 30 14:00:02.697759 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.697738    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" event={"ID":"47367ecc-1223-4952-a0b8-986afe4951b9","Type":"ContainerStarted","Data":"fe87221390dd6fb591e330daeea980e2fa05b4155e23bc905414ba6f61b94e7d"}
Oct 30 14:00:02.699476 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.699453    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" event={"ID":"40ba6924-2d7d-41bc-9749-458e7a705e64","Type":"ContainerStarted","Data":"a77052963a1dcd269d973ec37f3e32abe2ba0ca54801a1cdc0cf4b87fe74cc66"}
Oct 30 14:00:02.701185 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.701146    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" event={"ID":"f764f2cc-15c4-460c-930c-9b16c9fa82ab","Type":"ContainerStarted","Data":"42357f4ced72d1b253dcd75d85c308aebb6c5a3cb247aab2d7a3d9ef4012e683"}
Oct 30 14:00:02.703127 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.703103    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" event={"ID":"077541bb-8a79-4080-8a2c-2644004dbe3d","Type":"ContainerStarted","Data":"1e39aba0db888dcac7f80904d6b50708d58ab95931537d4658482b07e4a01457"}
Oct 30 14:00:02.705139 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.704880    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerStarted","Data":"6afae7eb4b38019debddb5c241ecf815138063798023673f99a6da65c7c9b57c"}
Oct 30 14:00:02.717963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.717925    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" event={"ID":"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b","Type":"ContainerStarted","Data":"340f982a53bdd73e0ca30302d416944f015495659e14435ac0b4e50ff10900e3"}
Oct 30 14:00:02.720420 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.720392    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" event={"ID":"9a630f92-dc84-454b-8f37-c80cdcfe40b1","Type":"ContainerStarted","Data":"d7db5fce4a50fcc2f33733fed6f03580bbfa7572095984d428195726f23c4961"}
Oct 30 14:00:02.721902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.721875    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" event={"ID":"851ad965-5bcf-46ed-bdf4-1764a61b901a","Type":"ContainerStarted","Data":"40db06c64ff24b17966e1f2239bbbe97e0a96b1f564097f2a10f71f839b46f89"}
Oct 30 14:00:02.723765 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.723736    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerStarted","Data":"09f69b0faaccaa56f947baf73ecee45e8a96b6ce3f0b9d63d31d12a13008b460"}
Oct 30 14:00:02.723887 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.723817    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerStarted","Data":"612e75c27c996cbedf5e7ba8f30a116da9e45023fdeced9c6305d6246b94fffb"}
Oct 30 14:00:02.728043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.727994    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" podStartSLOduration=2.727976932 podStartE2EDuration="2.727976932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.725681579 +0000 UTC m=+3412.985919810" watchObservedRunningTime="2026-10-30 14:00:02.727976932 +0000 UTC m=+3412.988215168"
Oct 30 14:00:02.731986 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.731963    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerStarted","Data":"c886bf6a072d2b790d6b0e1f1a90488c6637f52bc5e202b3346a29c47eebb84a"}
Oct 30 14:00:02.734254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.734231    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" event={"ID":"2804875f-4c19-409b-8646-c17b8507f6f2","Type":"ContainerStarted","Data":"dad9960baed21831b000e11b66d7137dff3f8d65910941ec87ae6d7e1613567b"}
Oct 30 14:00:02.736241 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.736215    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" event={"ID":"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0","Type":"ContainerStarted","Data":"0267f1defe3cc65bfda8bee020fb68a1e37c4f34d2039939a300e5ab2ffcefa9"}
Oct 30 14:00:02.738017 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.737988    4291 generic.go:356] "Generic (PLEG): container finished" podID="421d7f0d-0d50-4985-95f4-4058c691b840" containerID="dcd982b060f364dc84d5ebf1dc105ad5b28b3c094d84d8d720b5051734c29680" exitCode=60
Oct 30 14:00:02.738140 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.738037    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerDied","Data":"dcd982b060f364dc84d5ebf1dc105ad5b28b3c094d84d8d720b5051734c29680"}
Oct 30 14:00:02.738450 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.738435    4291 scope.go:122] "RemoveContainer" containerID="dcd982b060f364dc84d5ebf1dc105ad5b28b3c094d84d8d720b5051734c29680"
Oct 30 14:00:02.804155 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.804104    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" podStartSLOduration=2.804090154 podStartE2EDuration="2.804090154s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.800541743 +0000 UTC m=+3413.060779976" watchObservedRunningTime="2026-10-30 14:00:02.804090154 +0000 UTC m=+3413.064328388"
Oct 30 14:00:02.840147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.840052    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" podStartSLOduration=2.8400378 podStartE2EDuration="2.8400378s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.837001847 +0000 UTC m=+3413.097240080" watchObservedRunningTime="2026-10-30 14:00:02.8400378 +0000 UTC m=+3413.100276033"
Oct 30 14:00:02.882172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.881611    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" podStartSLOduration=2.881592504 podStartE2EDuration="2.881592504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.875616909 +0000 UTC m=+3413.135855146" watchObservedRunningTime="2026-10-30 14:00:02.881592504 +0000 UTC m=+3413.141830738"
Oct 30 14:00:02.916075 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.916013    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" podStartSLOduration=2.915998448 podStartE2EDuration="2.915998448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.912447586 +0000 UTC m=+3413.172685837" watchObservedRunningTime="2026-10-30 14:00:02.915998448 +0000 UTC m=+3413.176236682"
Oct 30 14:00:02.966048 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:02.964999    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" podStartSLOduration=2.964979726 podStartE2EDuration="2.964979726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:02.958822402 +0000 UTC m=+3413.219060638" watchObservedRunningTime="2026-10-30 14:00:02.964979726 +0000 UTC m=+3413.225217961"
Oct 30 14:00:03.003498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.003443    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" podStartSLOduration=3.003428932 podStartE2EDuration="3.003428932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.002463552 +0000 UTC m=+3413.262701786" watchObservedRunningTime="2026-10-30 14:00:03.003428932 +0000 UTC m=+3413.263667165"
Oct 30 14:00:03.056379 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.056329    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" podStartSLOduration=3.056313187 podStartE2EDuration="3.056313187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.054769887 +0000 UTC m=+3413.315008121" watchObservedRunningTime="2026-10-30 14:00:03.056313187 +0000 UTC m=+3413.316551421"
Oct 30 14:00:03.100565 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.100513    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" podStartSLOduration=3.100497938 podStartE2EDuration="3.100497938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.096923955 +0000 UTC m=+3413.357162183" watchObservedRunningTime="2026-10-30 14:00:03.100497938 +0000 UTC m=+3413.360736173"
Oct 30 14:00:03.135310 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.135255    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" podStartSLOduration=3.135242536 podStartE2EDuration="3.135242536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.134012311 +0000 UTC m=+3413.394250544" watchObservedRunningTime="2026-10-30 14:00:03.135242536 +0000 UTC m=+3413.395480769"
Oct 30 14:00:03.759835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.759771    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" event={"ID":"851ad965-5bcf-46ed-bdf4-1764a61b901a","Type":"ContainerStarted","Data":"5c28c03dee4414fce7eaff14d1a339220ab1360a7f8d1718f0acd2155030321e"}
Oct 30 14:00:03.763115 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.763080    4291 generic.go:356] "Generic (PLEG): container finished" podID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerID="09f69b0faaccaa56f947baf73ecee45e8a96b6ce3f0b9d63d31d12a13008b460" exitCode=60
Oct 30 14:00:03.763304 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.763167    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerDied","Data":"09f69b0faaccaa56f947baf73ecee45e8a96b6ce3f0b9d63d31d12a13008b460"}
Oct 30 14:00:03.763505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.763489    4291 scope.go:122] "RemoveContainer" containerID="09f69b0faaccaa56f947baf73ecee45e8a96b6ce3f0b9d63d31d12a13008b460"
Oct 30 14:00:03.765776 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.765691    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerStarted","Data":"502c287af9311c2b7af13f5f3177a22ee517ae401ed18981823a745e2c1c913a"}
Oct 30 14:00:03.768507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.768465    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" event={"ID":"2804875f-4c19-409b-8646-c17b8507f6f2","Type":"ContainerStarted","Data":"79b522bee96e01c9afa2f03c3c7f812791ff12eb69c8ee80132a05f519573ef1"}
Oct 30 14:00:03.786710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.771168    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" event={"ID":"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0","Type":"ContainerStarted","Data":"0f729f717714286ce749d7b3661597c09d8fe34360ed89279bcbb1c9dfec61b3"}
Oct 30 14:00:03.786710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.774191    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerStarted","Data":"fb05844fcf7ff1183f4474c2c89cdfddcdaf5b3e4a6857304d40c7a9dcff6c4b"}
Oct 30 14:00:03.786710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.779319    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerStarted","Data":"371398949e113377624007a3b7327bf16905ad7e104b44e6af1d1c9d81f41cf1"}
Oct 30 14:00:03.786710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.784561    4291 generic.go:356] "Generic (PLEG): container finished" podID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerID="a151398c75e0c9d713202f1131dd676463743ba30913422b0fa39baf9ee4c200" exitCode=60
Oct 30 14:00:03.786710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.784701    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" event={"ID":"02c6941b-9f8e-4ca3-84cc-7fa377906a40","Type":"ContainerDied","Data":"a151398c75e0c9d713202f1131dd676463743ba30913422b0fa39baf9ee4c200"}
Oct 30 14:00:03.786710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.785284    4291 scope.go:122] "RemoveContainer" containerID="a151398c75e0c9d713202f1131dd676463743ba30913422b0fa39baf9ee4c200"
Oct 30 14:00:03.788358 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.788187    4291 generic.go:356] "Generic (PLEG): container finished" podID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerID="8f2b69b6b0f0307b55a7092d5f48f48d016b370a55bc82b49dc596d196369441" exitCode=60
Oct 30 14:00:03.788358 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.788226    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4" event={"ID":"5b5fc286-5675-4c60-bb29-9b0bf79bd701","Type":"ContainerDied","Data":"8f2b69b6b0f0307b55a7092d5f48f48d016b370a55bc82b49dc596d196369441"}
Oct 30 14:00:03.788719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.788696    4291 scope.go:122] "RemoveContainer" containerID="8f2b69b6b0f0307b55a7092d5f48f48d016b370a55bc82b49dc596d196369441"
Oct 30 14:00:03.794706 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.794629    4291 generic.go:356] "Generic (PLEG): container finished" podID="cc3d3284-1405-4e29-91fd-fba47df039df" containerID="e0b8ff8fa7d7e0c52d4090e9f2fda0d3ce14d805092f6c4d1aee5470f61ec3a6" exitCode=60
Oct 30 14:00:03.794990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.794692    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" event={"ID":"cc3d3284-1405-4e29-91fd-fba47df039df","Type":"ContainerDied","Data":"e0b8ff8fa7d7e0c52d4090e9f2fda0d3ce14d805092f6c4d1aee5470f61ec3a6"}
Oct 30 14:00:03.795187 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.795165    4291 scope.go:122] "RemoveContainer" containerID="e0b8ff8fa7d7e0c52d4090e9f2fda0d3ce14d805092f6c4d1aee5470f61ec3a6"
Oct 30 14:00:03.797269 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.797245    4291 generic.go:356] "Generic (PLEG): container finished" podID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerID="1a5ee8ce0c3b864f4177e723899afa37f63049a8ac3d9e8cde0ca65302b6632a" exitCode=60
Oct 30 14:00:03.797372 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.797319    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" event={"ID":"e43b7f3d-c901-4f59-a039-04c48e60d984","Type":"ContainerDied","Data":"1a5ee8ce0c3b864f4177e723899afa37f63049a8ac3d9e8cde0ca65302b6632a"}
Oct 30 14:00:03.797609 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.797592    4291 scope.go:122] "RemoveContainer" containerID="1a5ee8ce0c3b864f4177e723899afa37f63049a8ac3d9e8cde0ca65302b6632a"
Oct 30 14:00:03.799289 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.799268    4291 generic.go:356] "Generic (PLEG): container finished" podID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerID="a04255785fc734007fc6e1994fc45a5db63baff9be329299691bd0578d984c87" exitCode=60
Oct 30 14:00:03.799398 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.799380    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" event={"ID":"5d440f70-f544-432c-a8f1-e38d9cb7ec68","Type":"ContainerDied","Data":"a04255785fc734007fc6e1994fc45a5db63baff9be329299691bd0578d984c87"}
Oct 30 14:00:03.799912 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.799895    4291 scope.go:122] "RemoveContainer" containerID="a04255785fc734007fc6e1994fc45a5db63baff9be329299691bd0578d984c87"
Oct 30 14:00:03.809784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.809719    4291 generic.go:356] "Generic (PLEG): container finished" podID="addb6a6e-6673-4881-b84c-fa9561235a47" containerID="9e4adf242e78e599a7f537df4b9b9f1a89b8b60aa7b1803fda80413dc70f7eef" exitCode=60
Oct 30 14:00:03.809882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.809809    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" event={"ID":"addb6a6e-6673-4881-b84c-fa9561235a47","Type":"ContainerDied","Data":"9e4adf242e78e599a7f537df4b9b9f1a89b8b60aa7b1803fda80413dc70f7eef"}
Oct 30 14:00:03.809882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.809862    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" event={"ID":"addb6a6e-6673-4881-b84c-fa9561235a47","Type":"ContainerStarted","Data":"568a744d1696e61a17528c44527f5468d83c217242a9295b1c2a57891fc4659b"}
Oct 30 14:00:03.810125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.810087    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" podStartSLOduration=3.81007599 podStartE2EDuration="3.81007599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.801241044 +0000 UTC m=+3414.061479278" watchObservedRunningTime="2026-10-30 14:00:03.81007599 +0000 UTC m=+3414.070314610"
Oct 30 14:00:03.815068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.815037    4291 generic.go:356] "Generic (PLEG): container finished" podID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerID="e030a969bb21be2fd96ca1cbd80b45b4ff65448eff2208e0bbb454374aba0b62" exitCode=60
Oct 30 14:00:03.815215 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.815160    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerDied","Data":"e030a969bb21be2fd96ca1cbd80b45b4ff65448eff2208e0bbb454374aba0b62"}
Oct 30 14:00:03.815441 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.815422    4291 scope.go:122] "RemoveContainer" containerID="e030a969bb21be2fd96ca1cbd80b45b4ff65448eff2208e0bbb454374aba0b62"
Oct 30 14:00:03.817267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.817242    4291 generic.go:356] "Generic (PLEG): container finished" podID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerID="8ed5433b2cf5b8802c2bc76f0eab7286b4d430d7253391697aae56a750a85c5e" exitCode=60
Oct 30 14:00:03.817477 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.817458    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" event={"ID":"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0","Type":"ContainerDied","Data":"8ed5433b2cf5b8802c2bc76f0eab7286b4d430d7253391697aae56a750a85c5e"}
Oct 30 14:00:03.818083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.817581    4291 scope.go:122] "RemoveContainer" containerID="8ed5433b2cf5b8802c2bc76f0eab7286b4d430d7253391697aae56a750a85c5e"
Oct 30 14:00:03.823068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.823043    4291 generic.go:356] "Generic (PLEG): container finished" podID="47367ecc-1223-4952-a0b8-986afe4951b9" containerID="3596d2add8fa42942ead1e5033901024db21838030e64d53b0f290c133c9ba04" exitCode=60
Oct 30 14:00:03.823158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.823140    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" event={"ID":"47367ecc-1223-4952-a0b8-986afe4951b9","Type":"ContainerDied","Data":"3596d2add8fa42942ead1e5033901024db21838030e64d53b0f290c133c9ba04"}
Oct 30 14:00:03.823441 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.823421    4291 scope.go:122] "RemoveContainer" containerID="3596d2add8fa42942ead1e5033901024db21838030e64d53b0f290c133c9ba04"
Oct 30 14:00:03.826593 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.826571    4291 generic.go:356] "Generic (PLEG): container finished" podID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerID="3fa790b0581b943de9ebbfddefdeb50ce51a5668db496c7e8f2bf5374b4f7d7d" exitCode=60
Oct 30 14:00:03.826742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.826723    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" event={"ID":"40ba6924-2d7d-41bc-9749-458e7a705e64","Type":"ContainerDied","Data":"3fa790b0581b943de9ebbfddefdeb50ce51a5668db496c7e8f2bf5374b4f7d7d"}
Oct 30 14:00:03.826940 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.826925    4291 scope.go:122] "RemoveContainer" containerID="3fa790b0581b943de9ebbfddefdeb50ce51a5668db496c7e8f2bf5374b4f7d7d"
Oct 30 14:00:03.828782 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.828760    4291 generic.go:356] "Generic (PLEG): container finished" podID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerID="42357f4ced72d1b253dcd75d85c308aebb6c5a3cb247aab2d7a3d9ef4012e683" exitCode=60
Oct 30 14:00:03.828869 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.828841    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" event={"ID":"f764f2cc-15c4-460c-930c-9b16c9fa82ab","Type":"ContainerDied","Data":"42357f4ced72d1b253dcd75d85c308aebb6c5a3cb247aab2d7a3d9ef4012e683"}
Oct 30 14:00:03.829039 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.829011    4291 scope.go:122] "RemoveContainer" containerID="42357f4ced72d1b253dcd75d85c308aebb6c5a3cb247aab2d7a3d9ef4012e683"
Oct 30 14:00:03.830919 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.830900    4291 generic.go:356] "Generic (PLEG): container finished" podID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerID="c33c337af20e131c3581e064219ae5387b096bd581bd8868e52060f87ebc037c" exitCode=60
Oct 30 14:00:03.831023 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.830982    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" event={"ID":"077541bb-8a79-4080-8a2c-2644004dbe3d","Type":"ContainerDied","Data":"c33c337af20e131c3581e064219ae5387b096bd581bd8868e52060f87ebc037c"}
Oct 30 14:00:03.831186 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.831169    4291 scope.go:122] "RemoveContainer" containerID="c33c337af20e131c3581e064219ae5387b096bd581bd8868e52060f87ebc037c"
Oct 30 14:00:03.834445 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.834422    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerStarted","Data":"6992bebeefe81ea3835342593cbc80d6c23c82a81a82c8fc8df178e2a6004661"}
Oct 30 14:00:03.837412 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.837386    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" event={"ID":"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b","Type":"ContainerStarted","Data":"5c63210f4490f47c3f802b87c3e81e55a1530c599ebc7bbe37ae9356fd0d5690"}
Oct 30 14:00:03.839286 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.839265    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" event={"ID":"9a630f92-dc84-454b-8f37-c80cdcfe40b1","Type":"ContainerStarted","Data":"2345d1ec826cd5e25b0bb2751ba8c0abc0c0938eecb0c1dd675f68221a890bcc"}
Oct 30 14:00:03.989434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.989384    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" podStartSLOduration=3.989367505 podStartE2EDuration="3.989367505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.893914433 +0000 UTC m=+3414.154152668" watchObservedRunningTime="2026-10-30 14:00:03.989367505 +0000 UTC m=+3414.249605810"
Oct 30 14:00:03.994307 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:03.994261    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" podStartSLOduration=3.994247859 podStartE2EDuration="3.994247859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:03.991499017 +0000 UTC m=+3414.251737249" watchObservedRunningTime="2026-10-30 14:00:03.994247859 +0000 UTC m=+3414.254486093"
Oct 30 14:00:04.072621 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.072571    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" podStartSLOduration=4.072553101 podStartE2EDuration="4.072553101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:04.06741213 +0000 UTC m=+3414.327650367" watchObservedRunningTime="2026-10-30 14:00:04.072553101 +0000 UTC m=+3414.332791332"
Oct 30 14:00:04.104340 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.104292    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" podStartSLOduration=4.104278642 podStartE2EDuration="4.104278642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:04.102738031 +0000 UTC m=+3414.362976263" watchObservedRunningTime="2026-10-30 14:00:04.104278642 +0000 UTC m=+3414.364516886"
Oct 30 14:00:04.335805 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.335756    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" podStartSLOduration=4.335741714 podStartE2EDuration="4.335741714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:04.331407455 +0000 UTC m=+3414.591645690" watchObservedRunningTime="2026-10-30 14:00:04.335741714 +0000 UTC m=+3414.595979948"
Oct 30 14:00:04.409684 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.409591    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" podStartSLOduration=4.409576743 podStartE2EDuration="4.409576743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:04.406482845 +0000 UTC m=+3414.666721079" watchObservedRunningTime="2026-10-30 14:00:04.409576743 +0000 UTC m=+3414.669815010"
Oct 30 14:00:04.448884 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.448829    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" podStartSLOduration=4.448815568 podStartE2EDuration="4.448815568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:04.444570037 +0000 UTC m=+3414.704808271" watchObservedRunningTime="2026-10-30 14:00:04.448815568 +0000 UTC m=+3414.709053801"
Oct 30 14:00:04.593746 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.593693    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" podStartSLOduration=4.593679601 podStartE2EDuration="4.593679601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:04.592538971 +0000 UTC m=+3414.852777204" watchObservedRunningTime="2026-10-30 14:00:04.593679601 +0000 UTC m=+3414.853917835"
Oct 30 14:00:04.852563 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.852522    4291 generic.go:356] "Generic (PLEG): container finished" podID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerID="6992bebeefe81ea3835342593cbc80d6c23c82a81a82c8fc8df178e2a6004661" exitCode=60
Oct 30 14:00:04.853305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.852615    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerDied","Data":"6992bebeefe81ea3835342593cbc80d6c23c82a81a82c8fc8df178e2a6004661"}
Oct 30 14:00:04.853305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.852948    4291 scope.go:122] "RemoveContainer" containerID="6992bebeefe81ea3835342593cbc80d6c23c82a81a82c8fc8df178e2a6004661"
Oct 30 14:00:04.854846 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.854822    4291 generic.go:356] "Generic (PLEG): container finished" podID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerID="5c63210f4490f47c3f802b87c3e81e55a1530c599ebc7bbe37ae9356fd0d5690" exitCode=60
Oct 30 14:00:04.854983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.854929    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" event={"ID":"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b","Type":"ContainerDied","Data":"5c63210f4490f47c3f802b87c3e81e55a1530c599ebc7bbe37ae9356fd0d5690"}
Oct 30 14:00:04.855270 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.855208    4291 scope.go:122] "RemoveContainer" containerID="5c63210f4490f47c3f802b87c3e81e55a1530c599ebc7bbe37ae9356fd0d5690"
Oct 30 14:00:04.857055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.857032    4291 generic.go:356] "Generic (PLEG): container finished" podID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerID="2345d1ec826cd5e25b0bb2751ba8c0abc0c0938eecb0c1dd675f68221a890bcc" exitCode=60
Oct 30 14:00:04.857311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.857085    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" event={"ID":"9a630f92-dc84-454b-8f37-c80cdcfe40b1","Type":"ContainerDied","Data":"2345d1ec826cd5e25b0bb2751ba8c0abc0c0938eecb0c1dd675f68221a890bcc"}
Oct 30 14:00:04.857437 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.857407    4291 scope.go:122] "RemoveContainer" containerID="2345d1ec826cd5e25b0bb2751ba8c0abc0c0938eecb0c1dd675f68221a890bcc"
Oct 30 14:00:04.861699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.859837    4291 generic.go:356] "Generic (PLEG): container finished" podID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerID="5c28c03dee4414fce7eaff14d1a339220ab1360a7f8d1718f0acd2155030321e" exitCode=60
Oct 30 14:00:04.861699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.859933    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" event={"ID":"851ad965-5bcf-46ed-bdf4-1764a61b901a","Type":"ContainerDied","Data":"5c28c03dee4414fce7eaff14d1a339220ab1360a7f8d1718f0acd2155030321e"}
Oct 30 14:00:04.861699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.860186    4291 scope.go:122] "RemoveContainer" containerID="5c28c03dee4414fce7eaff14d1a339220ab1360a7f8d1718f0acd2155030321e"
Oct 30 14:00:04.862257 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.862237    4291 generic.go:356] "Generic (PLEG): container finished" podID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerID="502c287af9311c2b7af13f5f3177a22ee517ae401ed18981823a745e2c1c913a" exitCode=60
Oct 30 14:00:04.862338 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.862313    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerDied","Data":"502c287af9311c2b7af13f5f3177a22ee517ae401ed18981823a745e2c1c913a"}
Oct 30 14:00:04.862545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.862528    4291 scope.go:122] "RemoveContainer" containerID="502c287af9311c2b7af13f5f3177a22ee517ae401ed18981823a745e2c1c913a"
Oct 30 14:00:04.864748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.864725    4291 generic.go:356] "Generic (PLEG): container finished" podID="2804875f-4c19-409b-8646-c17b8507f6f2" containerID="79b522bee96e01c9afa2f03c3c7f812791ff12eb69c8ee80132a05f519573ef1" exitCode=60
Oct 30 14:00:04.864859 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.864810    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" event={"ID":"2804875f-4c19-409b-8646-c17b8507f6f2","Type":"ContainerDied","Data":"79b522bee96e01c9afa2f03c3c7f812791ff12eb69c8ee80132a05f519573ef1"}
Oct 30 14:00:04.865020 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.865001    4291 scope.go:122] "RemoveContainer" containerID="79b522bee96e01c9afa2f03c3c7f812791ff12eb69c8ee80132a05f519573ef1"
Oct 30 14:00:04.866586 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.866566    4291 generic.go:356] "Generic (PLEG): container finished" podID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerID="0f729f717714286ce749d7b3661597c09d8fe34360ed89279bcbb1c9dfec61b3" exitCode=60
Oct 30 14:00:04.866721 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.866618    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" event={"ID":"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0","Type":"ContainerDied","Data":"0f729f717714286ce749d7b3661597c09d8fe34360ed89279bcbb1c9dfec61b3"}
Oct 30 14:00:04.866839 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.866824    4291 scope.go:122] "RemoveContainer" containerID="0f729f717714286ce749d7b3661597c09d8fe34360ed89279bcbb1c9dfec61b3"
Oct 30 14:00:04.878696 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.878667    4291 generic.go:356] "Generic (PLEG): container finished" podID="421d7f0d-0d50-4985-95f4-4058c691b840" containerID="fb05844fcf7ff1183f4474c2c89cdfddcdaf5b3e4a6857304d40c7a9dcff6c4b" exitCode=60
Oct 30 14:00:04.878831 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.878803    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerDied","Data":"fb05844fcf7ff1183f4474c2c89cdfddcdaf5b3e4a6857304d40c7a9dcff6c4b"}
Oct 30 14:00:04.879017 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.878991    4291 scope.go:122] "RemoveContainer" containerID="dcd982b060f364dc84d5ebf1dc105ad5b28b3c094d84d8d720b5051734c29680"
Oct 30 14:00:04.879436 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.879408    4291 scope.go:122] "RemoveContainer" containerID="fb05844fcf7ff1183f4474c2c89cdfddcdaf5b3e4a6857304d40c7a9dcff6c4b"
Oct 30 14:00:04.879849 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:04.879820    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv_odh-ai-gateway-infra(421d7f0d-0d50-4985-95f4-4058c691b840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" podUID="421d7f0d-0d50-4985-95f4-4058c691b840"
Oct 30 14:00:04.881831 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.881791    4291 generic.go:356] "Generic (PLEG): container finished" podID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerID="371398949e113377624007a3b7327bf16905ad7e104b44e6af1d1c9d81f41cf1" exitCode=60
Oct 30 14:00:04.882010 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.881957    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerDied","Data":"371398949e113377624007a3b7327bf16905ad7e104b44e6af1d1c9d81f41cf1"}
Oct 30 14:00:04.882167 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.882150    4291 scope.go:122] "RemoveContainer" containerID="371398949e113377624007a3b7327bf16905ad7e104b44e6af1d1c9d81f41cf1"
Oct 30 14:00:04.885250 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.885225    4291 generic.go:356] "Generic (PLEG): container finished" podID="addb6a6e-6673-4881-b84c-fa9561235a47" containerID="568a744d1696e61a17528c44527f5468d83c217242a9295b1c2a57891fc4659b" exitCode=60
Oct 30 14:00:04.885356 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.885282    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" event={"ID":"addb6a6e-6673-4881-b84c-fa9561235a47","Type":"ContainerDied","Data":"568a744d1696e61a17528c44527f5468d83c217242a9295b1c2a57891fc4659b"}
Oct 30 14:00:04.885575 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:04.885560    4291 scope.go:122] "RemoveContainer" containerID="568a744d1696e61a17528c44527f5468d83c217242a9295b1c2a57891fc4659b"
Oct 30 14:00:04.885905 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:04.885885    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4_odh-ai-gateway-infra(addb6a6e-6673-4881-b84c-fa9561235a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" podUID="addb6a6e-6673-4881-b84c-fa9561235a47"
Oct 30 14:00:05.128428 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.128402    4291 scope.go:122] "RemoveContainer" containerID="9e4adf242e78e599a7f537df4b9b9f1a89b8b60aa7b1803fda80413dc70f7eef"
Oct 30 14:00:05.909118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.909035    4291 scope.go:122] "RemoveContainer" containerID="568a744d1696e61a17528c44527f5468d83c217242a9295b1c2a57891fc4659b"
Oct 30 14:00:05.910021 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:05.909988    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4_odh-ai-gateway-infra(addb6a6e-6673-4881-b84c-fa9561235a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" podUID="addb6a6e-6673-4881-b84c-fa9561235a47"
Oct 30 14:00:05.917551 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.917521    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerStarted","Data":"8dc9b0e26d75e6a7f2191b7990f31c8d31f55dd60f69846edaa86cffb9bba98f"}
Oct 30 14:00:05.920516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.920489    4291 generic.go:356] "Generic (PLEG): container finished" podID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerID="6f6e5890c15e39c9c568b2dde795cf79e9b6d3569235cc2e02a69f7d74ca7c11" exitCode=60
Oct 30 14:00:05.920610 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.920568    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" event={"ID":"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0","Type":"ContainerDied","Data":"6f6e5890c15e39c9c568b2dde795cf79e9b6d3569235cc2e02a69f7d74ca7c11"}
Oct 30 14:00:05.920689 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.920644    4291 scope.go:122] "RemoveContainer" containerID="8ed5433b2cf5b8802c2bc76f0eab7286b4d430d7253391697aae56a750a85c5e"
Oct 30 14:00:05.920858 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.920840    4291 scope.go:122] "RemoveContainer" containerID="6f6e5890c15e39c9c568b2dde795cf79e9b6d3569235cc2e02a69f7d74ca7c11"
Oct 30 14:00:05.921098 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:05.921078    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5_odh-ai-gateway-infra(bf93e80a-eb0c-447c-a9c6-a37be5c77ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0"
Oct 30 14:00:05.922608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.922587    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" event={"ID":"47367ecc-1223-4952-a0b8-986afe4951b9","Type":"ContainerStarted","Data":"8f3f3637e222fb878d23b912ff3fb2482f5bb650fc75b13d7354512edb40d083"}
Oct 30 14:00:05.924618 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.924595    4291 generic.go:356] "Generic (PLEG): container finished" podID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerID="920aacfe34023a37e92ec728de7dab15c1e21a0a8b90714e5ea57e4bd50a459f" exitCode=60
Oct 30 14:00:05.924723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.924687    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" event={"ID":"40ba6924-2d7d-41bc-9749-458e7a705e64","Type":"ContainerDied","Data":"920aacfe34023a37e92ec728de7dab15c1e21a0a8b90714e5ea57e4bd50a459f"}
Oct 30 14:00:05.925232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.924919    4291 scope.go:122] "RemoveContainer" containerID="920aacfe34023a37e92ec728de7dab15c1e21a0a8b90714e5ea57e4bd50a459f"
Oct 30 14:00:05.925232 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:05.925189    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc_odh-ai-gateway-infra(40ba6924-2d7d-41bc-9749-458e7a705e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64"
Oct 30 14:00:05.926886 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.926860    4291 generic.go:356] "Generic (PLEG): container finished" podID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerID="94e30ae236f35cc889435021a1520a1333b48ce189269a0674bdb1b0d59cfe21" exitCode=60
Oct 30 14:00:05.926984 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.926943    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" event={"ID":"f764f2cc-15c4-460c-930c-9b16c9fa82ab","Type":"ContainerDied","Data":"94e30ae236f35cc889435021a1520a1333b48ce189269a0674bdb1b0d59cfe21"}
Oct 30 14:00:05.927177 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.927146    4291 scope.go:122] "RemoveContainer" containerID="94e30ae236f35cc889435021a1520a1333b48ce189269a0674bdb1b0d59cfe21"
Oct 30 14:00:05.927458 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:05.927434    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj_odh-ai-gateway-infra(f764f2cc-15c4-460c-930c-9b16c9fa82ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab"
Oct 30 14:00:05.929227 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.929207    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" event={"ID":"077541bb-8a79-4080-8a2c-2644004dbe3d","Type":"ContainerStarted","Data":"0b46032e526b9f45284002f91a343108f5fc5254c8b2271201c7dbb2b628664d"}
Oct 30 14:00:05.931311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.931290    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerStarted","Data":"4cb8e0cac053d24d9c5331f759978129efbb774d0dcc78692923dbdeaed61673"}
Oct 30 14:00:05.933092 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.933072    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" event={"ID":"9a630f92-dc84-454b-8f37-c80cdcfe40b1","Type":"ContainerStarted","Data":"d0eb86cf395d8ba783672111f87e1319dd956866585898015355400db5248e2f"}
Oct 30 14:00:05.934911 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.934888    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" event={"ID":"851ad965-5bcf-46ed-bdf4-1764a61b901a","Type":"ContainerStarted","Data":"df1f89e38ecd301c3dde00511098a83f153f641415ed79cc5afc0ad3cd7e6f34"}
Oct 30 14:00:05.936717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.936696    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerStarted","Data":"41586aa39229c3d878e1a44100ef1fbd5d4e15564eb2048265b89b45f3807a95"}
Oct 30 14:00:05.938531 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.938510    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerStarted","Data":"4d9d9aac4b6ca33d5886a3cf37fe4a7c9c2d5504e6178867a5ceafcb07b5b5f6"}
Oct 30 14:00:05.940305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.940286    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" event={"ID":"2804875f-4c19-409b-8646-c17b8507f6f2","Type":"ContainerStarted","Data":"41eb4f2eb12770da9b14754bd7eff922210e5bd06023c1641ae0cc3ccda6bd5e"}
Oct 30 14:00:05.942260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.942242    4291 scope.go:122] "RemoveContainer" containerID="fb05844fcf7ff1183f4474c2c89cdfddcdaf5b3e4a6857304d40c7a9dcff6c4b"
Oct 30 14:00:05.942469 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:05.942451    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv_odh-ai-gateway-infra(421d7f0d-0d50-4985-95f4-4058c691b840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" podUID="421d7f0d-0d50-4985-95f4-4058c691b840"
Oct 30 14:00:05.943784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.943764    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerStarted","Data":"f0787b38fe2fdad762c9235f56ba19bdb383bbded81f14fd604ca2599d85985b"}
Oct 30 14:00:05.945682 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.945657    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" event={"ID":"02c6941b-9f8e-4ca3-84cc-7fa377906a40","Type":"ContainerStarted","Data":"8055e4a7ad5724e8fc991bb356889dfd2de6765108e5a137bac22cc691b391c9"}
Oct 30 14:00:05.947689 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.947667    4291 generic.go:356] "Generic (PLEG): container finished" podID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerID="a7de37b701e13a55aeeeba38052f5340a107ce8062b1d67dd721265f56ec983c" exitCode=60
Oct 30 14:00:05.947797 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.947753    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4" event={"ID":"5b5fc286-5675-4c60-bb29-9b0bf79bd701","Type":"ContainerDied","Data":"a7de37b701e13a55aeeeba38052f5340a107ce8062b1d67dd721265f56ec983c"}
Oct 30 14:00:05.947951 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.947934    4291 scope.go:122] "RemoveContainer" containerID="a7de37b701e13a55aeeeba38052f5340a107ce8062b1d67dd721265f56ec983c"
Oct 30 14:00:05.948190 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:05.948165    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4_odh-ai-gateway-infra(5b5fc286-5675-4c60-bb29-9b0bf79bd701)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701"
Oct 30 14:00:05.950058 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.950035    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" event={"ID":"cc3d3284-1405-4e29-91fd-fba47df039df","Type":"ContainerStarted","Data":"5b76972c6606a17ce98a8814eacc02f3c64eab2978ab061f7c18c870f0a8986b"}
Oct 30 14:00:05.953221 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.953198    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" event={"ID":"e43b7f3d-c901-4f59-a039-04c48e60d984","Type":"ContainerStarted","Data":"757602a0d88ddcf9dd23264cca972c20651761c997c860dc386fe6da0fa03270"}
Oct 30 14:00:05.958770 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.955785    4291 generic.go:356] "Generic (PLEG): container finished" podID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerID="7a0bdb437ee0ba39c54a1a20fda64a272721f88c51fe00710590404a4d8f995a" exitCode=60
Oct 30 14:00:05.958770 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.955837    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" event={"ID":"5d440f70-f544-432c-a8f1-e38d9cb7ec68","Type":"ContainerDied","Data":"7a0bdb437ee0ba39c54a1a20fda64a272721f88c51fe00710590404a4d8f995a"}
Oct 30 14:00:05.958770 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:05.956099    4291 scope.go:122] "RemoveContainer" containerID="7a0bdb437ee0ba39c54a1a20fda64a272721f88c51fe00710590404a4d8f995a"
Oct 30 14:00:05.958770 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:05.956339    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz_odh-ai-gateway-infra(5d440f70-f544-432c-a8f1-e38d9cb7ec68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68"
Oct 30 14:00:06.070910 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.070860    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" podStartSLOduration=6.070846412 podStartE2EDuration="6.070846412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:06.068132956 +0000 UTC m=+3416.328371187" watchObservedRunningTime="2026-10-30 14:00:06.070846412 +0000 UTC m=+3416.331084646"
Oct 30 14:00:06.966991 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.966961    4291 generic.go:356] "Generic (PLEG): container finished" podID="cc3d3284-1405-4e29-91fd-fba47df039df" containerID="5b76972c6606a17ce98a8814eacc02f3c64eab2978ab061f7c18c870f0a8986b" exitCode=60
Oct 30 14:00:06.967403 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.967046    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" event={"ID":"cc3d3284-1405-4e29-91fd-fba47df039df","Type":"ContainerDied","Data":"5b76972c6606a17ce98a8814eacc02f3c64eab2978ab061f7c18c870f0a8986b"}
Oct 30 14:00:06.967403 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.967369    4291 scope.go:122] "RemoveContainer" containerID="5b76972c6606a17ce98a8814eacc02f3c64eab2978ab061f7c18c870f0a8986b"
Oct 30 14:00:06.967720 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.967698    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x_odh-ai-gateway-infra(cc3d3284-1405-4e29-91fd-fba47df039df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" podUID="cc3d3284-1405-4e29-91fd-fba47df039df"
Oct 30 14:00:06.969361 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.969335    4291 generic.go:356] "Generic (PLEG): container finished" podID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerID="757602a0d88ddcf9dd23264cca972c20651761c997c860dc386fe6da0fa03270" exitCode=60
Oct 30 14:00:06.969451 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.969416    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" event={"ID":"e43b7f3d-c901-4f59-a039-04c48e60d984","Type":"ContainerDied","Data":"757602a0d88ddcf9dd23264cca972c20651761c997c860dc386fe6da0fa03270"}
Oct 30 14:00:06.969719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.969697    4291 scope.go:122] "RemoveContainer" containerID="757602a0d88ddcf9dd23264cca972c20651761c997c860dc386fe6da0fa03270"
Oct 30 14:00:06.970013 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.969993    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg_odh-ai-gateway-infra(e43b7f3d-c901-4f59-a039-04c48e60d984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984"
Oct 30 14:00:06.971322 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.971300    4291 generic.go:356] "Generic (PLEG): container finished" podID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerID="8dc9b0e26d75e6a7f2191b7990f31c8d31f55dd60f69846edaa86cffb9bba98f" exitCode=60
Oct 30 14:00:06.971407 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.971347    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerDied","Data":"8dc9b0e26d75e6a7f2191b7990f31c8d31f55dd60f69846edaa86cffb9bba98f"}
Oct 30 14:00:06.971611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.971595    4291 scope.go:122] "RemoveContainer" containerID="8dc9b0e26d75e6a7f2191b7990f31c8d31f55dd60f69846edaa86cffb9bba98f"
Oct 30 14:00:06.971941 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.971908    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6_odh-ai-gateway-infra(398137cb-99db-4931-bca0-f6e1b1b931a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7"
Oct 30 14:00:06.973262 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.973243    4291 generic.go:356] "Generic (PLEG): container finished" podID="47367ecc-1223-4952-a0b8-986afe4951b9" containerID="8f3f3637e222fb878d23b912ff3fb2482f5bb650fc75b13d7354512edb40d083" exitCode=60
Oct 30 14:00:06.973373 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.973325    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" event={"ID":"47367ecc-1223-4952-a0b8-986afe4951b9","Type":"ContainerDied","Data":"8f3f3637e222fb878d23b912ff3fb2482f5bb650fc75b13d7354512edb40d083"}
Oct 30 14:00:06.973571 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.973554    4291 scope.go:122] "RemoveContainer" containerID="8f3f3637e222fb878d23b912ff3fb2482f5bb650fc75b13d7354512edb40d083"
Oct 30 14:00:06.973918 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.973892    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx_odh-ai-gateway-infra(47367ecc-1223-4952-a0b8-986afe4951b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" podUID="47367ecc-1223-4952-a0b8-986afe4951b9"
Oct 30 14:00:06.975260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.975240    4291 generic.go:356] "Generic (PLEG): container finished" podID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerID="0b46032e526b9f45284002f91a343108f5fc5254c8b2271201c7dbb2b628664d" exitCode=60
Oct 30 14:00:06.975358 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.975310    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" event={"ID":"077541bb-8a79-4080-8a2c-2644004dbe3d","Type":"ContainerDied","Data":"0b46032e526b9f45284002f91a343108f5fc5254c8b2271201c7dbb2b628664d"}
Oct 30 14:00:06.975623 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.975602    4291 scope.go:122] "RemoveContainer" containerID="0b46032e526b9f45284002f91a343108f5fc5254c8b2271201c7dbb2b628664d"
Oct 30 14:00:06.976400 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.976378    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t_odh-ai-gateway-infra(077541bb-8a79-4080-8a2c-2644004dbe3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d"
Oct 30 14:00:06.977308 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.977289    4291 generic.go:356] "Generic (PLEG): container finished" podID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerID="4cb8e0cac053d24d9c5331f759978129efbb774d0dcc78692923dbdeaed61673" exitCode=60
Oct 30 14:00:06.977395 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.977370    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerDied","Data":"4cb8e0cac053d24d9c5331f759978129efbb774d0dcc78692923dbdeaed61673"}
Oct 30 14:00:06.977648 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.977570    4291 scope.go:122] "RemoveContainer" containerID="4cb8e0cac053d24d9c5331f759978129efbb774d0dcc78692923dbdeaed61673"
Oct 30 14:00:06.977845 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.977824    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92_odh-ai-gateway-infra(f5fb8d13-b0d8-439d-b7b2-9227b69604e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7"
Oct 30 14:00:06.979294 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.979275    4291 generic.go:356] "Generic (PLEG): container finished" podID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerID="91db4841d357c894bb50166600f33ac40fb467bf47ac53a66cb04056434634a9" exitCode=60
Oct 30 14:00:06.979381 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.979349    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" event={"ID":"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b","Type":"ContainerDied","Data":"91db4841d357c894bb50166600f33ac40fb467bf47ac53a66cb04056434634a9"}
Oct 30 14:00:06.979543 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.979527    4291 scope.go:122] "RemoveContainer" containerID="91db4841d357c894bb50166600f33ac40fb467bf47ac53a66cb04056434634a9"
Oct 30 14:00:06.979821 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.979787    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr_odh-ai-gateway-infra(4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b"
Oct 30 14:00:06.981326 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.981305    4291 generic.go:356] "Generic (PLEG): container finished" podID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerID="d0eb86cf395d8ba783672111f87e1319dd956866585898015355400db5248e2f" exitCode=60
Oct 30 14:00:06.981409 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.981395    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" event={"ID":"9a630f92-dc84-454b-8f37-c80cdcfe40b1","Type":"ContainerDied","Data":"d0eb86cf395d8ba783672111f87e1319dd956866585898015355400db5248e2f"}
Oct 30 14:00:06.981588 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.981573    4291 scope.go:122] "RemoveContainer" containerID="d0eb86cf395d8ba783672111f87e1319dd956866585898015355400db5248e2f"
Oct 30 14:00:06.981862 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.981843    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889480-hkr66_odh-ai-gateway-infra(9a630f92-dc84-454b-8f37-c80cdcfe40b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1"
Oct 30 14:00:06.983328 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.983307    4291 generic.go:356] "Generic (PLEG): container finished" podID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerID="df1f89e38ecd301c3dde00511098a83f153f641415ed79cc5afc0ad3cd7e6f34" exitCode=60
Oct 30 14:00:06.983399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.983378    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" event={"ID":"851ad965-5bcf-46ed-bdf4-1764a61b901a","Type":"ContainerDied","Data":"df1f89e38ecd301c3dde00511098a83f153f641415ed79cc5afc0ad3cd7e6f34"}
Oct 30 14:00:06.983557 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.983543    4291 scope.go:122] "RemoveContainer" containerID="df1f89e38ecd301c3dde00511098a83f153f641415ed79cc5afc0ad3cd7e6f34"
Oct 30 14:00:06.983788 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.983770    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j_odh-ai-gateway-infra(851ad965-5bcf-46ed-bdf4-1764a61b901a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a"
Oct 30 14:00:06.985180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.985155    4291 generic.go:356] "Generic (PLEG): container finished" podID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerID="41586aa39229c3d878e1a44100ef1fbd5d4e15564eb2048265b89b45f3807a95" exitCode=60
Oct 30 14:00:06.985297 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.985239    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerDied","Data":"41586aa39229c3d878e1a44100ef1fbd5d4e15564eb2048265b89b45f3807a95"}
Oct 30 14:00:06.985477 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.985462    4291 scope.go:122] "RemoveContainer" containerID="41586aa39229c3d878e1a44100ef1fbd5d4e15564eb2048265b89b45f3807a95"
Oct 30 14:00:06.985726 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.985708    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd_odh-ai-gateway-infra(d0e7264f-66cf-45f6-91db-4126497c2ae6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6"
Oct 30 14:00:06.987043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.987027    4291 generic.go:356] "Generic (PLEG): container finished" podID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerID="4d9d9aac4b6ca33d5886a3cf37fe4a7c9c2d5504e6178867a5ceafcb07b5b5f6" exitCode=60
Oct 30 14:00:06.987106 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.987093    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerDied","Data":"4d9d9aac4b6ca33d5886a3cf37fe4a7c9c2d5504e6178867a5ceafcb07b5b5f6"}
Oct 30 14:00:06.987283 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.987269    4291 scope.go:122] "RemoveContainer" containerID="4d9d9aac4b6ca33d5886a3cf37fe4a7c9c2d5504e6178867a5ceafcb07b5b5f6"
Oct 30 14:00:06.987490 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.987466    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp_odh-ai-gateway-infra(b4fb266d-2bcf-4120-b82e-1f8ebefee7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6"
Oct 30 14:00:06.989851 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.989830    4291 generic.go:356] "Generic (PLEG): container finished" podID="2804875f-4c19-409b-8646-c17b8507f6f2" containerID="41eb4f2eb12770da9b14754bd7eff922210e5bd06023c1641ae0cc3ccda6bd5e" exitCode=60
Oct 30 14:00:06.989940 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.989904    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" event={"ID":"2804875f-4c19-409b-8646-c17b8507f6f2","Type":"ContainerDied","Data":"41eb4f2eb12770da9b14754bd7eff922210e5bd06023c1641ae0cc3ccda6bd5e"}
Oct 30 14:00:06.990096 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.990079    4291 scope.go:122] "RemoveContainer" containerID="41eb4f2eb12770da9b14754bd7eff922210e5bd06023c1641ae0cc3ccda6bd5e"
Oct 30 14:00:06.990323 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.990304    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56_odh-ai-gateway-infra(2804875f-4c19-409b-8646-c17b8507f6f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" podUID="2804875f-4c19-409b-8646-c17b8507f6f2"
Oct 30 14:00:06.991796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.991777    4291 generic.go:356] "Generic (PLEG): container finished" podID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerID="1466030b862aede4985289bae1f216b573594b3c4b2bae03e9218c4b59ce6bf0" exitCode=60
Oct 30 14:00:06.991876 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.991852    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" event={"ID":"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0","Type":"ContainerDied","Data":"1466030b862aede4985289bae1f216b573594b3c4b2bae03e9218c4b59ce6bf0"}
Oct 30 14:00:06.992072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.992053    4291 scope.go:122] "RemoveContainer" containerID="1466030b862aede4985289bae1f216b573594b3c4b2bae03e9218c4b59ce6bf0"
Oct 30 14:00:06.992298 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.992276    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r_odh-ai-gateway-infra(d9af2f8f-927e-4948-bf5a-3bcb8afd89b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0"
Oct 30 14:00:06.993710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.993692    4291 generic.go:356] "Generic (PLEG): container finished" podID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerID="f0787b38fe2fdad762c9235f56ba19bdb383bbded81f14fd604ca2599d85985b" exitCode=60
Oct 30 14:00:06.993802 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.993770    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerDied","Data":"f0787b38fe2fdad762c9235f56ba19bdb383bbded81f14fd604ca2599d85985b"}
Oct 30 14:00:06.993985 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.993967    4291 scope.go:122] "RemoveContainer" containerID="f0787b38fe2fdad762c9235f56ba19bdb383bbded81f14fd604ca2599d85985b"
Oct 30 14:00:06.994201 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.994181    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx_odh-ai-gateway-infra(8f297ee2-5036-4ac9-8839-c80d2eb293e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5"
Oct 30 14:00:06.995786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.995760    4291 generic.go:356] "Generic (PLEG): container finished" podID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerID="8055e4a7ad5724e8fc991bb356889dfd2de6765108e5a137bac22cc691b391c9" exitCode=60
Oct 30 14:00:06.995870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.995828    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" event={"ID":"02c6941b-9f8e-4ca3-84cc-7fa377906a40","Type":"ContainerDied","Data":"8055e4a7ad5724e8fc991bb356889dfd2de6765108e5a137bac22cc691b391c9"}
Oct 30 14:00:06.996095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.996079    4291 scope.go:122] "RemoveContainer" containerID="8055e4a7ad5724e8fc991bb356889dfd2de6765108e5a137bac22cc691b391c9"
Oct 30 14:00:06.996224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.996205    4291 scope.go:122] "RemoveContainer" containerID="7a0bdb437ee0ba39c54a1a20fda64a272721f88c51fe00710590404a4d8f995a"
Oct 30 14:00:06.996303 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.996235    4291 scope.go:122] "RemoveContainer" containerID="6f6e5890c15e39c9c568b2dde795cf79e9b6d3569235cc2e02a69f7d74ca7c11"
Oct 30 14:00:06.996303 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.996262    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts_odh-ai-gateway-infra(02c6941b-9f8e-4ca3-84cc-7fa377906a40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40"
Oct 30 14:00:06.996387 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.996367    4291 scope.go:122] "RemoveContainer" containerID="920aacfe34023a37e92ec728de7dab15c1e21a0a8b90714e5ea57e4bd50a459f"
Oct 30 14:00:06.996464 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.996450    4291 scope.go:122] "RemoveContainer" containerID="a7de37b701e13a55aeeeba38052f5340a107ce8062b1d67dd721265f56ec983c"
Oct 30 14:00:06.996524 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.996467    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz_odh-ai-gateway-infra(5d440f70-f544-432c-a8f1-e38d9cb7ec68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68"
Oct 30 14:00:06.996524 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.996511    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5_odh-ai-gateway-infra(bf93e80a-eb0c-447c-a9c6-a37be5c77ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0"
Oct 30 14:00:06.996613 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:06.996524    4291 scope.go:122] "RemoveContainer" containerID="94e30ae236f35cc889435021a1520a1333b48ce189269a0674bdb1b0d59cfe21"
Oct 30 14:00:06.996672 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.996626    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc_odh-ai-gateway-infra(40ba6924-2d7d-41bc-9749-458e7a705e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64"
Oct 30 14:00:06.996672 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.996629    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4_odh-ai-gateway-infra(5b5fc286-5675-4c60-bb29-9b0bf79bd701)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701"
Oct 30 14:00:06.996766 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:06.996750    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj_odh-ai-gateway-infra(f764f2cc-15c4-460c-930c-9b16c9fa82ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab"
Oct 30 14:00:07.023852 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.023819    4291 scope.go:122] "RemoveContainer" containerID="3fa790b0581b943de9ebbfddefdeb50ce51a5668db496c7e8f2bf5374b4f7d7d"
Oct 30 14:00:07.039495 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.039473    4291 scope.go:122] "RemoveContainer" containerID="42357f4ced72d1b253dcd75d85c308aebb6c5a3cb247aab2d7a3d9ef4012e683"
Oct 30 14:00:07.056030 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.056010    4291 scope.go:122] "RemoveContainer" containerID="8f2b69b6b0f0307b55a7092d5f48f48d016b370a55bc82b49dc596d196369441"
Oct 30 14:00:07.070200 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.070179    4291 scope.go:122] "RemoveContainer" containerID="a04255785fc734007fc6e1994fc45a5db63baff9be329299691bd0578d984c87"
Oct 30 14:00:07.084191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.084163    4291 scope.go:122] "RemoveContainer" containerID="e0b8ff8fa7d7e0c52d4090e9f2fda0d3ce14d805092f6c4d1aee5470f61ec3a6"
Oct 30 14:00:07.099759 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.099737    4291 scope.go:122] "RemoveContainer" containerID="1a5ee8ce0c3b864f4177e723899afa37f63049a8ac3d9e8cde0ca65302b6632a"
Oct 30 14:00:07.113582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.113563    4291 scope.go:122] "RemoveContainer" containerID="e030a969bb21be2fd96ca1cbd80b45b4ff65448eff2208e0bbb454374aba0b62"
Oct 30 14:00:07.127878 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.127859    4291 scope.go:122] "RemoveContainer" containerID="3596d2add8fa42942ead1e5033901024db21838030e64d53b0f290c133c9ba04"
Oct 30 14:00:07.142049 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.142029    4291 scope.go:122] "RemoveContainer" containerID="c33c337af20e131c3581e064219ae5387b096bd581bd8868e52060f87ebc037c"
Oct 30 14:00:07.155479 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.155457    4291 scope.go:122] "RemoveContainer" containerID="6992bebeefe81ea3835342593cbc80d6c23c82a81a82c8fc8df178e2a6004661"
Oct 30 14:00:07.169179 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.169160    4291 scope.go:122] "RemoveContainer" containerID="5c63210f4490f47c3f802b87c3e81e55a1530c599ebc7bbe37ae9356fd0d5690"
Oct 30 14:00:07.183024 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.183000    4291 scope.go:122] "RemoveContainer" containerID="2345d1ec826cd5e25b0bb2751ba8c0abc0c0938eecb0c1dd675f68221a890bcc"
Oct 30 14:00:07.198166 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.198146    4291 scope.go:122] "RemoveContainer" containerID="5c28c03dee4414fce7eaff14d1a339220ab1360a7f8d1718f0acd2155030321e"
Oct 30 14:00:07.213981 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.213964    4291 scope.go:122] "RemoveContainer" containerID="09f69b0faaccaa56f947baf73ecee45e8a96b6ce3f0b9d63d31d12a13008b460"
Oct 30 14:00:07.228584 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.228553    4291 scope.go:122] "RemoveContainer" containerID="502c287af9311c2b7af13f5f3177a22ee517ae401ed18981823a745e2c1c913a"
Oct 30 14:00:07.241994 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.241974    4291 scope.go:122] "RemoveContainer" containerID="79b522bee96e01c9afa2f03c3c7f812791ff12eb69c8ee80132a05f519573ef1"
Oct 30 14:00:07.262100 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.262080    4291 scope.go:122] "RemoveContainer" containerID="0f729f717714286ce749d7b3661597c09d8fe34360ed89279bcbb1c9dfec61b3"
Oct 30 14:00:07.276240 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.276221    4291 scope.go:122] "RemoveContainer" containerID="371398949e113377624007a3b7327bf16905ad7e104b44e6af1d1c9d81f41cf1"
Oct 30 14:00:07.290037 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:07.290020    4291 scope.go:122] "RemoveContainer" containerID="a151398c75e0c9d713202f1131dd676463743ba30913422b0fa39baf9ee4c200"
Oct 30 14:00:08.018147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.018108    4291 scope.go:122] "RemoveContainer" containerID="0b46032e526b9f45284002f91a343108f5fc5254c8b2271201c7dbb2b628664d"
Oct 30 14:00:08.018609 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.018443    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t_odh-ai-gateway-infra(077541bb-8a79-4080-8a2c-2644004dbe3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d"
Oct 30 14:00:08.019794 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.019771    4291 scope.go:122] "RemoveContainer" containerID="4cb8e0cac053d24d9c5331f759978129efbb774d0dcc78692923dbdeaed61673"
Oct 30 14:00:08.020115 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.020091    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92_odh-ai-gateway-infra(f5fb8d13-b0d8-439d-b7b2-9227b69604e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7"
Oct 30 14:00:08.021292 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.021270    4291 scope.go:122] "RemoveContainer" containerID="91db4841d357c894bb50166600f33ac40fb467bf47ac53a66cb04056434634a9"
Oct 30 14:00:08.021577 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.021552    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr_odh-ai-gateway-infra(4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b"
Oct 30 14:00:08.022878 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.022859    4291 scope.go:122] "RemoveContainer" containerID="d0eb86cf395d8ba783672111f87e1319dd956866585898015355400db5248e2f"
Oct 30 14:00:08.023143 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.023124    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889480-hkr66_odh-ai-gateway-infra(9a630f92-dc84-454b-8f37-c80cdcfe40b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1"
Oct 30 14:00:08.024417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.024399    4291 scope.go:122] "RemoveContainer" containerID="df1f89e38ecd301c3dde00511098a83f153f641415ed79cc5afc0ad3cd7e6f34"
Oct 30 14:00:08.024707 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.024686    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j_odh-ai-gateway-infra(851ad965-5bcf-46ed-bdf4-1764a61b901a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a"
Oct 30 14:00:08.025920 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.025904    4291 scope.go:122] "RemoveContainer" containerID="41586aa39229c3d878e1a44100ef1fbd5d4e15564eb2048265b89b45f3807a95"
Oct 30 14:00:08.026172 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.026151    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd_odh-ai-gateway-infra(d0e7264f-66cf-45f6-91db-4126497c2ae6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6"
Oct 30 14:00:08.027511 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.027499    4291 scope.go:122] "RemoveContainer" containerID="4d9d9aac4b6ca33d5886a3cf37fe4a7c9c2d5504e6178867a5ceafcb07b5b5f6"
Oct 30 14:00:08.027768 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.027744    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp_odh-ai-gateway-infra(b4fb266d-2bcf-4120-b82e-1f8ebefee7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6"
Oct 30 14:00:08.029078 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.029061    4291 scope.go:122] "RemoveContainer" containerID="41eb4f2eb12770da9b14754bd7eff922210e5bd06023c1641ae0cc3ccda6bd5e"
Oct 30 14:00:08.029307 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.029288    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56_odh-ai-gateway-infra(2804875f-4c19-409b-8646-c17b8507f6f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" podUID="2804875f-4c19-409b-8646-c17b8507f6f2"
Oct 30 14:00:08.030544 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.030526    4291 scope.go:122] "RemoveContainer" containerID="1466030b862aede4985289bae1f216b573594b3c4b2bae03e9218c4b59ce6bf0"
Oct 30 14:00:08.030785 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.030763    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r_odh-ai-gateway-infra(d9af2f8f-927e-4948-bf5a-3bcb8afd89b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0"
Oct 30 14:00:08.032162 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.032126    4291 scope.go:122] "RemoveContainer" containerID="f0787b38fe2fdad762c9235f56ba19bdb383bbded81f14fd604ca2599d85985b"
Oct 30 14:00:08.032383 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.032364    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx_odh-ai-gateway-infra(8f297ee2-5036-4ac9-8839-c80d2eb293e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5"
Oct 30 14:00:08.033725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.033698    4291 scope.go:122] "RemoveContainer" containerID="8055e4a7ad5724e8fc991bb356889dfd2de6765108e5a137bac22cc691b391c9"
Oct 30 14:00:08.033944 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.033923    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts_odh-ai-gateway-infra(02c6941b-9f8e-4ca3-84cc-7fa377906a40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40"
Oct 30 14:00:08.036599 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.036581    4291 scope.go:122] "RemoveContainer" containerID="5b76972c6606a17ce98a8814eacc02f3c64eab2978ab061f7c18c870f0a8986b"
Oct 30 14:00:08.036902 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.036839    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x_odh-ai-gateway-infra(cc3d3284-1405-4e29-91fd-fba47df039df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" podUID="cc3d3284-1405-4e29-91fd-fba47df039df"
Oct 30 14:00:08.038165 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.038146    4291 scope.go:122] "RemoveContainer" containerID="757602a0d88ddcf9dd23264cca972c20651761c997c860dc386fe6da0fa03270"
Oct 30 14:00:08.038359 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.038342    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg_odh-ai-gateway-infra(e43b7f3d-c901-4f59-a039-04c48e60d984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984"
Oct 30 14:00:08.041011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.040995    4291 scope.go:122] "RemoveContainer" containerID="8dc9b0e26d75e6a7f2191b7990f31c8d31f55dd60f69846edaa86cffb9bba98f"
Oct 30 14:00:08.041254 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.041236    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6_odh-ai-gateway-infra(398137cb-99db-4931-bca0-f6e1b1b931a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7"
Oct 30 14:00:08.043933 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:08.043915    4291 scope.go:122] "RemoveContainer" containerID="8f3f3637e222fb878d23b912ff3fb2482f5bb650fc75b13d7354512edb40d083"
Oct 30 14:00:08.044123 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.044105    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx_odh-ai-gateway-infra(47367ecc-1223-4952-a0b8-986afe4951b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" podUID="47367ecc-1223-4952-a0b8-986afe4951b9"
Oct 30 14:00:08.212430 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:08.212393    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:00:09.183910 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:09.183871    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"]
Oct 30 14:00:12.999409 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:12.999372    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"]
Oct 30 14:00:13.409891 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.409837    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:13.490502 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.490467    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b5fc286-5675-4c60-bb29-9b0bf79bd701-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5fc286-5675-4c60-bb29-9b0bf79bd701-maas-api-service-ca\") pod \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") "
Oct 30 14:00:13.491145 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.491114    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b5fc286-5675-4c60-bb29-9b0bf79bd701-kube-api-access-jg262\" (UniqueName: \"kubernetes.io/projected/5b5fc286-5675-4c60-bb29-9b0bf79bd701-kube-api-access-jg262\") pod \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\" (UID: \"5b5fc286-5675-4c60-bb29-9b0bf79bd701\") "
Oct 30 14:00:13.506623 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.491685    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b5fc286-5675-4c60-bb29-9b0bf79bd701-maas-api-service-ca" pod "5b5fc286-5675-4c60-bb29-9b0bf79bd701" (UID: "5b5fc286-5675-4c60-bb29-9b0bf79bd701"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:13.506847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.506186    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b5fc286-5675-4c60-bb29-9b0bf79bd701-kube-api-access-jg262" pod "5b5fc286-5675-4c60-bb29-9b0bf79bd701" (UID: "5b5fc286-5675-4c60-bb29-9b0bf79bd701"). InnerVolumeSpecName "kube-api-access-jg262". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:13.608464 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.608434    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg262\" (UniqueName: \"kubernetes.io/projected/5b5fc286-5675-4c60-bb29-9b0bf79bd701-kube-api-access-jg262\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:13.608464 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.608463    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b5fc286-5675-4c60-bb29-9b0bf79bd701-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:13.733199 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.733146    4291 scope.go:122] "RemoveContainer" containerID="61faebd9c7ebeb1d9da6e6bb49f6feb8de2d456546d29f467566ee46a138c2b8"
Oct 30 14:00:13.747079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.747057    4291 scope.go:122] "RemoveContainer" containerID="d28facc1bfb1ab6bff5d51a0cd9b2710d86bfe521722026c08449c9e8c40d42b"
Oct 30 14:00:13.760190 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.760147    4291 scope.go:122] "RemoveContainer" containerID="82d16ece91e5e045d81c2ada06c0bd78977bc52c477c2c280cbdb882b6d2ab86"
Oct 30 14:00:13.952918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.952883    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"]
Oct 30 14:00:13.961373 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:13.961329    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-88352d-j7hg7"]
Oct 30 14:00:14.107356 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:14.107328    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"
Oct 30 14:00:14.107356 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:14.107342    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4" event={"ID":"5b5fc286-5675-4c60-bb29-9b0bf79bd701","Type":"ContainerDied","Data":"94f357dcfe58dcacca772fffbbf523421ad8db8d56d96c01289757d2ff47c1e6"}
Oct 30 14:00:14.107799 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:14.107432    4291 scope.go:122] "RemoveContainer" containerID="a7de37b701e13a55aeeeba38052f5340a107ce8062b1d67dd721265f56ec983c"
Oct 30 14:00:14.164297 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:14.164228    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"]
Oct 30 14:00:14.168806 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:14.168781    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-88352d-29889480-jjqk4"]
Oct 30 14:00:15.883248 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:15.883207    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1dc6b5a3-1dd6-487e-9bba-f63c844adf06" path="/var/lib/kubelet/pods/1dc6b5a3-1dd6-487e-9bba-f63c844adf06/volumes"
Oct 30 14:00:15.883928 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:15.883906    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" path="/var/lib/kubelet/pods/5b5fc286-5675-4c60-bb29-9b0bf79bd701/volumes"
Oct 30 14:00:16.871196 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:16.871165    4291 scope.go:122] "RemoveContainer" containerID="920aacfe34023a37e92ec728de7dab15c1e21a0a8b90714e5ea57e4bd50a459f"
Oct 30 14:00:16.871474 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:16.871449    4291 scope.go:122] "RemoveContainer" containerID="8f3f3637e222fb878d23b912ff3fb2482f5bb650fc75b13d7354512edb40d083"
Oct 30 14:00:16.871620 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:16.871599    4291 scope.go:122] "RemoveContainer" containerID="0b46032e526b9f45284002f91a343108f5fc5254c8b2271201c7dbb2b628664d"
Oct 30 14:00:16.872382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:16.871879    4291 scope.go:122] "RemoveContainer" containerID="fb05844fcf7ff1183f4474c2c89cdfddcdaf5b3e4a6857304d40c7a9dcff6c4b"
Oct 30 14:00:16.872382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:16.872051    4291 scope.go:122] "RemoveContainer" containerID="8055e4a7ad5724e8fc991bb356889dfd2de6765108e5a137bac22cc691b391c9"
Oct 30 14:00:17.871767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:17.871734    4291 scope.go:122] "RemoveContainer" containerID="41586aa39229c3d878e1a44100ef1fbd5d4e15564eb2048265b89b45f3807a95"
Oct 30 14:00:17.872235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:17.871948    4291 scope.go:122] "RemoveContainer" containerID="1466030b862aede4985289bae1f216b573594b3c4b2bae03e9218c4b59ce6bf0"
Oct 30 14:00:17.872235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:17.872057    4291 scope.go:122] "RemoveContainer" containerID="4d9d9aac4b6ca33d5886a3cf37fe4a7c9c2d5504e6178867a5ceafcb07b5b5f6"
Oct 30 14:00:17.872235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:17.872201    4291 scope.go:122] "RemoveContainer" containerID="f0787b38fe2fdad762c9235f56ba19bdb383bbded81f14fd604ca2599d85985b"
Oct 30 14:00:18.173516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:18.173486    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerStarted","Data":"efc27b2089127888519f58c98aadbd49fab6b7a7a8846ef9aa6a77ae716a1f7c"}
Oct 30 14:00:18.175548 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:18.175526    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" event={"ID":"02c6941b-9f8e-4ca3-84cc-7fa377906a40","Type":"ContainerStarted","Data":"3e1a6bcf2fd69482a6910cb0dbdf2b8ae79566e39103ac826541e1d56c4a443f"}
Oct 30 14:00:18.177493 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:18.177472    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" event={"ID":"47367ecc-1223-4952-a0b8-986afe4951b9","Type":"ContainerStarted","Data":"36ace568ca786eab8c2e3164efbc3d3f2b35784b759aa6123ff55cdf2e422915"}
Oct 30 14:00:18.179669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:18.179648    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" event={"ID":"40ba6924-2d7d-41bc-9749-458e7a705e64","Type":"ContainerStarted","Data":"80c04f3db21066bb54e4ac8af6fa3494d8387af865a5e9eeefbcbb8e3f524a72"}
Oct 30 14:00:18.181864 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:18.181840    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" event={"ID":"077541bb-8a79-4080-8a2c-2644004dbe3d","Type":"ContainerStarted","Data":"e72826d99e4299eb6984e3074a5f3d4fffb8bea49fae2035f05017c422c07259"}
Oct 30 14:00:18.872096 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:18.872062    4291 scope.go:122] "RemoveContainer" containerID="757602a0d88ddcf9dd23264cca972c20651761c997c860dc386fe6da0fa03270"
Oct 30 14:00:18.872608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:18.872297    4291 scope.go:122] "RemoveContainer" containerID="568a744d1696e61a17528c44527f5468d83c217242a9295b1c2a57891fc4659b"
Oct 30 14:00:19.192380 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.192344    4291 generic.go:356] "Generic (PLEG): container finished" podID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerID="80c04f3db21066bb54e4ac8af6fa3494d8387af865a5e9eeefbcbb8e3f524a72" exitCode=60
Oct 30 14:00:19.192544 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.192422    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" event={"ID":"40ba6924-2d7d-41bc-9749-458e7a705e64","Type":"ContainerDied","Data":"80c04f3db21066bb54e4ac8af6fa3494d8387af865a5e9eeefbcbb8e3f524a72"}
Oct 30 14:00:19.192544 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.192480    4291 scope.go:122] "RemoveContainer" containerID="920aacfe34023a37e92ec728de7dab15c1e21a0a8b90714e5ea57e4bd50a459f"
Oct 30 14:00:19.193022 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.192996    4291 scope.go:122] "RemoveContainer" containerID="80c04f3db21066bb54e4ac8af6fa3494d8387af865a5e9eeefbcbb8e3f524a72"
Oct 30 14:00:19.193526 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:19.193423    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc_odh-ai-gateway-infra(40ba6924-2d7d-41bc-9749-458e7a705e64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64"
Oct 30 14:00:19.194514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.194481    4291 generic.go:356] "Generic (PLEG): container finished" podID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerID="e72826d99e4299eb6984e3074a5f3d4fffb8bea49fae2035f05017c422c07259" exitCode=60
Oct 30 14:00:19.194606 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.194541    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" event={"ID":"077541bb-8a79-4080-8a2c-2644004dbe3d","Type":"ContainerDied","Data":"e72826d99e4299eb6984e3074a5f3d4fffb8bea49fae2035f05017c422c07259"}
Oct 30 14:00:19.194990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.194974    4291 scope.go:122] "RemoveContainer" containerID="e72826d99e4299eb6984e3074a5f3d4fffb8bea49fae2035f05017c422c07259"
Oct 30 14:00:19.195304 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:19.195286    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t_odh-ai-gateway-infra(077541bb-8a79-4080-8a2c-2644004dbe3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d"
Oct 30 14:00:19.196649 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.196611    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerStarted","Data":"1b7eaafa7678001e2024b30be03e7503963dffe7fe8bd0d3f96d0843343adb73"}
Oct 30 14:00:19.198903 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.198881    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerStarted","Data":"1bf1e67a930cd0aea456f6227cb25bb54684db3b301fc75921367fa4d9e5b8b0"}
Oct 30 14:00:19.200852 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.200827    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" event={"ID":"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0","Type":"ContainerStarted","Data":"86a75d45dfb4587381380cec3f5e72d264161eb4ce3b89bf76672fd6377791e7"}
Oct 30 14:00:19.203156 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.203134    4291 generic.go:356] "Generic (PLEG): container finished" podID="421d7f0d-0d50-4985-95f4-4058c691b840" containerID="efc27b2089127888519f58c98aadbd49fab6b7a7a8846ef9aa6a77ae716a1f7c" exitCode=60
Oct 30 14:00:19.203232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.203211    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerDied","Data":"efc27b2089127888519f58c98aadbd49fab6b7a7a8846ef9aa6a77ae716a1f7c"}
Oct 30 14:00:19.203608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.203584    4291 scope.go:122] "RemoveContainer" containerID="efc27b2089127888519f58c98aadbd49fab6b7a7a8846ef9aa6a77ae716a1f7c"
Oct 30 14:00:19.204012 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:19.203988    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv_odh-ai-gateway-infra(421d7f0d-0d50-4985-95f4-4058c691b840)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" podUID="421d7f0d-0d50-4985-95f4-4058c691b840"
Oct 30 14:00:19.205455 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.205432    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerStarted","Data":"ce7fd100ecef92ac712d362e997a58eeb4a9428eaf75b8a11a37a3e78b294634"}
Oct 30 14:00:19.207630 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.207605    4291 generic.go:356] "Generic (PLEG): container finished" podID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerID="3e1a6bcf2fd69482a6910cb0dbdf2b8ae79566e39103ac826541e1d56c4a443f" exitCode=60
Oct 30 14:00:19.207834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.207816    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" event={"ID":"02c6941b-9f8e-4ca3-84cc-7fa377906a40","Type":"ContainerDied","Data":"3e1a6bcf2fd69482a6910cb0dbdf2b8ae79566e39103ac826541e1d56c4a443f"}
Oct 30 14:00:19.208269 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.208249    4291 scope.go:122] "RemoveContainer" containerID="3e1a6bcf2fd69482a6910cb0dbdf2b8ae79566e39103ac826541e1d56c4a443f"
Oct 30 14:00:19.208678 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:19.208571    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts_odh-ai-gateway-infra(02c6941b-9f8e-4ca3-84cc-7fa377906a40)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40"
Oct 30 14:00:19.209172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.209153    4291 scope.go:122] "RemoveContainer" containerID="0b46032e526b9f45284002f91a343108f5fc5254c8b2271201c7dbb2b628664d"
Oct 30 14:00:19.210417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.210396    4291 generic.go:356] "Generic (PLEG): container finished" podID="47367ecc-1223-4952-a0b8-986afe4951b9" containerID="36ace568ca786eab8c2e3164efbc3d3f2b35784b759aa6123ff55cdf2e422915" exitCode=60
Oct 30 14:00:19.210503 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.210427    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" event={"ID":"47367ecc-1223-4952-a0b8-986afe4951b9","Type":"ContainerDied","Data":"36ace568ca786eab8c2e3164efbc3d3f2b35784b759aa6123ff55cdf2e422915"}
Oct 30 14:00:19.210732 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.210715    4291 scope.go:122] "RemoveContainer" containerID="36ace568ca786eab8c2e3164efbc3d3f2b35784b759aa6123ff55cdf2e422915"
Oct 30 14:00:19.210982 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:19.210957    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx_odh-ai-gateway-infra(47367ecc-1223-4952-a0b8-986afe4951b9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" podUID="47367ecc-1223-4952-a0b8-986afe4951b9"
Oct 30 14:00:19.225703 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.225677    4291 scope.go:122] "RemoveContainer" containerID="fb05844fcf7ff1183f4474c2c89cdfddcdaf5b3e4a6857304d40c7a9dcff6c4b"
Oct 30 14:00:19.241226 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.241204    4291 scope.go:122] "RemoveContainer" containerID="8055e4a7ad5724e8fc991bb356889dfd2de6765108e5a137bac22cc691b391c9"
Oct 30 14:00:19.249506 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.249483    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"]
Oct 30 14:00:19.258311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.258291    4291 scope.go:122] "RemoveContainer" containerID="8f3f3637e222fb878d23b912ff3fb2482f5bb650fc75b13d7354512edb40d083"
Oct 30 14:00:19.279783 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.279752    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"]
Oct 30 14:00:19.328907 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.328825    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"]
Oct 30 14:00:19.379351 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.379321    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"]
Oct 30 14:00:19.421232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.421196    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"]
Oct 30 14:00:19.883001 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.882968    4291 scope.go:122] "RemoveContainer" containerID="94e30ae236f35cc889435021a1520a1333b48ce189269a0674bdb1b0d59cfe21"
Oct 30 14:00:19.883498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.883133    4291 scope.go:122] "RemoveContainer" containerID="7a0bdb437ee0ba39c54a1a20fda64a272721f88c51fe00710590404a4d8f995a"
Oct 30 14:00:19.883498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.883349    4291 scope.go:122] "RemoveContainer" containerID="4cb8e0cac053d24d9c5331f759978129efbb774d0dcc78692923dbdeaed61673"
Oct 30 14:00:19.883498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.883434    4291 scope.go:122] "RemoveContainer" containerID="5b76972c6606a17ce98a8814eacc02f3c64eab2978ab061f7c18c870f0a8986b"
Oct 30 14:00:19.884705 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.883892    4291 scope.go:122] "RemoveContainer" containerID="41eb4f2eb12770da9b14754bd7eff922210e5bd06023c1641ae0cc3ccda6bd5e"
Oct 30 14:00:19.884705 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.884012    4291 scope.go:122] "RemoveContainer" containerID="df1f89e38ecd301c3dde00511098a83f153f641415ed79cc5afc0ad3cd7e6f34"
Oct 30 14:00:19.884705 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.884222    4291 scope.go:122] "RemoveContainer" containerID="8dc9b0e26d75e6a7f2191b7990f31c8d31f55dd60f69846edaa86cffb9bba98f"
Oct 30 14:00:19.884705 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.884406    4291 scope.go:122] "RemoveContainer" containerID="6f6e5890c15e39c9c568b2dde795cf79e9b6d3569235cc2e02a69f7d74ca7c11"
Oct 30 14:00:19.972739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.972712    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"]
Oct 30 14:00:19.995357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:19.995325    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"]
Oct 30 14:00:20.242780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.241483    4291 generic.go:356] "Generic (PLEG): container finished" podID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerID="1b7eaafa7678001e2024b30be03e7503963dffe7fe8bd0d3f96d0843343adb73" exitCode=60
Oct 30 14:00:20.242780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.241517    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerDied","Data":"1b7eaafa7678001e2024b30be03e7503963dffe7fe8bd0d3f96d0843343adb73"}
Oct 30 14:00:20.242780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.241607    4291 scope.go:122] "RemoveContainer" containerID="41586aa39229c3d878e1a44100ef1fbd5d4e15564eb2048265b89b45f3807a95"
Oct 30 14:00:20.242780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.242062    4291 scope.go:122] "RemoveContainer" containerID="1b7eaafa7678001e2024b30be03e7503963dffe7fe8bd0d3f96d0843343adb73"
Oct 30 14:00:20.242780 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:20.242509    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd_odh-ai-gateway-infra(d0e7264f-66cf-45f6-91db-4126497c2ae6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6"
Oct 30 14:00:20.251454 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.251427    4291 generic.go:356] "Generic (PLEG): container finished" podID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerID="1bf1e67a930cd0aea456f6227cb25bb54684db3b301fc75921367fa4d9e5b8b0" exitCode=60
Oct 30 14:00:20.251592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.251514    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerDied","Data":"1bf1e67a930cd0aea456f6227cb25bb54684db3b301fc75921367fa4d9e5b8b0"}
Oct 30 14:00:20.252084 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.251925    4291 scope.go:122] "RemoveContainer" containerID="1bf1e67a930cd0aea456f6227cb25bb54684db3b301fc75921367fa4d9e5b8b0"
Oct 30 14:00:20.252358 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:20.252248    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp_odh-ai-gateway-infra(b4fb266d-2bcf-4120-b82e-1f8ebefee7a6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6"
Oct 30 14:00:20.254587 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.254569    4291 generic.go:356] "Generic (PLEG): container finished" podID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerID="86a75d45dfb4587381380cec3f5e72d264161eb4ce3b89bf76672fd6377791e7" exitCode=60
Oct 30 14:00:20.254816 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.254654    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" event={"ID":"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0","Type":"ContainerDied","Data":"86a75d45dfb4587381380cec3f5e72d264161eb4ce3b89bf76672fd6377791e7"}
Oct 30 14:00:20.255237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.255221    4291 scope.go:122] "RemoveContainer" containerID="86a75d45dfb4587381380cec3f5e72d264161eb4ce3b89bf76672fd6377791e7"
Oct 30 14:00:20.255649 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:20.255613    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r_odh-ai-gateway-infra(d9af2f8f-927e-4948-bf5a-3bcb8afd89b0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0"
Oct 30 14:00:20.262392 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.262370    4291 generic.go:356] "Generic (PLEG): container finished" podID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerID="ce7fd100ecef92ac712d362e997a58eeb4a9428eaf75b8a11a37a3e78b294634" exitCode=60
Oct 30 14:00:20.262481 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.262438    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerDied","Data":"ce7fd100ecef92ac712d362e997a58eeb4a9428eaf75b8a11a37a3e78b294634"}
Oct 30 14:00:20.262947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.262929    4291 scope.go:122] "RemoveContainer" containerID="ce7fd100ecef92ac712d362e997a58eeb4a9428eaf75b8a11a37a3e78b294634"
Oct 30 14:00:20.263984 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:20.263940    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx_odh-ai-gateway-infra(8f297ee2-5036-4ac9-8839-c80d2eb293e5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5"
Oct 30 14:00:20.273399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.273369    4291 generic.go:356] "Generic (PLEG): container finished" podID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerID="c5cf0136bb38683bb3a29d9b444d4f9a9d3b6701ec9d7d65c72d3a27164b3544" exitCode=60
Oct 30 14:00:20.273494 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.273460    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" event={"ID":"e43b7f3d-c901-4f59-a039-04c48e60d984","Type":"ContainerDied","Data":"c5cf0136bb38683bb3a29d9b444d4f9a9d3b6701ec9d7d65c72d3a27164b3544"}
Oct 30 14:00:20.273894 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.273876    4291 scope.go:122] "RemoveContainer" containerID="c5cf0136bb38683bb3a29d9b444d4f9a9d3b6701ec9d7d65c72d3a27164b3544"
Oct 30 14:00:20.274227 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:20.274195    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg_odh-ai-gateway-infra(e43b7f3d-c901-4f59-a039-04c48e60d984)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984"
Oct 30 14:00:20.279787 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.279759    4291 generic.go:356] "Generic (PLEG): container finished" podID="addb6a6e-6673-4881-b84c-fa9561235a47" containerID="4ff9e2f09e817fcf44ef473194bf3fd394662bf628a7b8d9c7cc16a45b9840cc" exitCode=60
Oct 30 14:00:20.279992 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.279791    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" event={"ID":"addb6a6e-6673-4881-b84c-fa9561235a47","Type":"ContainerDied","Data":"4ff9e2f09e817fcf44ef473194bf3fd394662bf628a7b8d9c7cc16a45b9840cc"}
Oct 30 14:00:20.280330 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.280313    4291 scope.go:122] "RemoveContainer" containerID="4ff9e2f09e817fcf44ef473194bf3fd394662bf628a7b8d9c7cc16a45b9840cc"
Oct 30 14:00:20.280611 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:20.280589    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4_odh-ai-gateway-infra(addb6a6e-6673-4881-b84c-fa9561235a47)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" podUID="addb6a6e-6673-4881-b84c-fa9561235a47"
Oct 30 14:00:20.282843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.282826    4291 scope.go:122] "RemoveContainer" containerID="4d9d9aac4b6ca33d5886a3cf37fe4a7c9c2d5504e6178867a5ceafcb07b5b5f6"
Oct 30 14:00:20.304348 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.304322    4291 scope.go:122] "RemoveContainer" containerID="1466030b862aede4985289bae1f216b573594b3c4b2bae03e9218c4b59ce6bf0"
Oct 30 14:00:20.327737 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.327715    4291 scope.go:122] "RemoveContainer" containerID="f0787b38fe2fdad762c9235f56ba19bdb383bbded81f14fd604ca2599d85985b"
Oct 30 14:00:20.354079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.354050    4291 scope.go:122] "RemoveContainer" containerID="757602a0d88ddcf9dd23264cca972c20651761c997c860dc386fe6da0fa03270"
Oct 30 14:00:20.362341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.362010    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"]
Oct 30 14:00:20.392758 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.392712    4291 scope.go:122] "RemoveContainer" containerID="568a744d1696e61a17528c44527f5468d83c217242a9295b1c2a57891fc4659b"
Oct 30 14:00:20.403736 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.403672    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"]
Oct 30 14:00:20.531023 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.530986    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"]
Oct 30 14:00:20.733257 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.733125    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"]
Oct 30 14:00:20.859474 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.859366    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:20.873659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.872442    4291 scope.go:122] "RemoveContainer" containerID="91db4841d357c894bb50166600f33ac40fb467bf47ac53a66cb04056434634a9"
Oct 30 14:00:20.894654 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.874801    4291 scope.go:122] "RemoveContainer" containerID="d0eb86cf395d8ba783672111f87e1319dd956866585898015355400db5248e2f"
Oct 30 14:00:20.894654 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.886810    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:20.907325 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:20.907293    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:21.007256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.004706    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47367ecc-1223-4952-a0b8-986afe4951b9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47367ecc-1223-4952-a0b8-986afe4951b9-maas-api-service-ca\") pod \"47367ecc-1223-4952-a0b8-986afe4951b9\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") "
Oct 30 14:00:21.007256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.004839    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/077541bb-8a79-4080-8a2c-2644004dbe3d-kube-api-access-6nrr2\" (UniqueName: \"kubernetes.io/projected/077541bb-8a79-4080-8a2c-2644004dbe3d-kube-api-access-6nrr2\") pod \"077541bb-8a79-4080-8a2c-2644004dbe3d\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") "
Oct 30 14:00:21.007256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.004879    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02c6941b-9f8e-4ca3-84cc-7fa377906a40-kube-api-access-ljsjc\" (UniqueName: \"kubernetes.io/projected/02c6941b-9f8e-4ca3-84cc-7fa377906a40-kube-api-access-ljsjc\") pod \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") "
Oct 30 14:00:21.007256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.004966    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02c6941b-9f8e-4ca3-84cc-7fa377906a40-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6941b-9f8e-4ca3-84cc-7fa377906a40-maas-api-service-ca\") pod \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\" (UID: \"02c6941b-9f8e-4ca3-84cc-7fa377906a40\") "
Oct 30 14:00:21.007256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.005161    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47367ecc-1223-4952-a0b8-986afe4951b9-kube-api-access-ngl9z\" (UniqueName: \"kubernetes.io/projected/47367ecc-1223-4952-a0b8-986afe4951b9-kube-api-access-ngl9z\") pod \"47367ecc-1223-4952-a0b8-986afe4951b9\" (UID: \"47367ecc-1223-4952-a0b8-986afe4951b9\") "
Oct 30 14:00:21.007256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.005277    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/077541bb-8a79-4080-8a2c-2644004dbe3d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/077541bb-8a79-4080-8a2c-2644004dbe3d-maas-api-service-ca\") pod \"077541bb-8a79-4080-8a2c-2644004dbe3d\" (UID: \"077541bb-8a79-4080-8a2c-2644004dbe3d\") "
Oct 30 14:00:21.007256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.006966    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/077541bb-8a79-4080-8a2c-2644004dbe3d-maas-api-service-ca" pod "077541bb-8a79-4080-8a2c-2644004dbe3d" (UID: "077541bb-8a79-4080-8a2c-2644004dbe3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.007771 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.007609    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c6941b-9f8e-4ca3-84cc-7fa377906a40-maas-api-service-ca" pod "02c6941b-9f8e-4ca3-84cc-7fa377906a40" (UID: "02c6941b-9f8e-4ca3-84cc-7fa377906a40"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.008024 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.007981    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47367ecc-1223-4952-a0b8-986afe4951b9-maas-api-service-ca" pod "47367ecc-1223-4952-a0b8-986afe4951b9" (UID: "47367ecc-1223-4952-a0b8-986afe4951b9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.010820 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.010787    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47367ecc-1223-4952-a0b8-986afe4951b9-kube-api-access-ngl9z" pod "47367ecc-1223-4952-a0b8-986afe4951b9" (UID: "47367ecc-1223-4952-a0b8-986afe4951b9"). InnerVolumeSpecName "kube-api-access-ngl9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.010946 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.010848    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/077541bb-8a79-4080-8a2c-2644004dbe3d-kube-api-access-6nrr2" pod "077541bb-8a79-4080-8a2c-2644004dbe3d" (UID: "077541bb-8a79-4080-8a2c-2644004dbe3d"). InnerVolumeSpecName "kube-api-access-6nrr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.010946 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.010875    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c6941b-9f8e-4ca3-84cc-7fa377906a40-kube-api-access-ljsjc" pod "02c6941b-9f8e-4ca3-84cc-7fa377906a40" (UID: "02c6941b-9f8e-4ca3-84cc-7fa377906a40"). InnerVolumeSpecName "kube-api-access-ljsjc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.055454 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.055424    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:21.092776 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.092751    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:21.107145 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.107113    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/077541bb-8a79-4080-8a2c-2644004dbe3d-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.107145 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.107142    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47367ecc-1223-4952-a0b8-986afe4951b9-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.107312 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.107153    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nrr2\" (UniqueName: \"kubernetes.io/projected/077541bb-8a79-4080-8a2c-2644004dbe3d-kube-api-access-6nrr2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.107312 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.107163    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljsjc\" (UniqueName: \"kubernetes.io/projected/02c6941b-9f8e-4ca3-84cc-7fa377906a40-kube-api-access-ljsjc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.107312 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.107171    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02c6941b-9f8e-4ca3-84cc-7fa377906a40-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.107312 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.107180    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngl9z\" (UniqueName: \"kubernetes.io/projected/47367ecc-1223-4952-a0b8-986afe4951b9-kube-api-access-ngl9z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.208389 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.208304    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421d7f0d-0d50-4985-95f4-4058c691b840-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421d7f0d-0d50-4985-95f4-4058c691b840-maas-api-service-ca\") pod \"421d7f0d-0d50-4985-95f4-4058c691b840\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") "
Oct 30 14:00:21.208389 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.208347    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40ba6924-2d7d-41bc-9749-458e7a705e64-kube-api-access-s5l8z\" (UniqueName: \"kubernetes.io/projected/40ba6924-2d7d-41bc-9749-458e7a705e64-kube-api-access-s5l8z\") pod \"40ba6924-2d7d-41bc-9749-458e7a705e64\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") "
Oct 30 14:00:21.208691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.208426    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421d7f0d-0d50-4985-95f4-4058c691b840-kube-api-access-ndh64\" (UniqueName: \"kubernetes.io/projected/421d7f0d-0d50-4985-95f4-4058c691b840-kube-api-access-ndh64\") pod \"421d7f0d-0d50-4985-95f4-4058c691b840\" (UID: \"421d7f0d-0d50-4985-95f4-4058c691b840\") "
Oct 30 14:00:21.208764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.208693    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40ba6924-2d7d-41bc-9749-458e7a705e64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ba6924-2d7d-41bc-9749-458e7a705e64-maas-api-service-ca\") pod \"40ba6924-2d7d-41bc-9749-458e7a705e64\" (UID: \"40ba6924-2d7d-41bc-9749-458e7a705e64\") "
Oct 30 14:00:21.208764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.208733    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421d7f0d-0d50-4985-95f4-4058c691b840-maas-api-service-ca" pod "421d7f0d-0d50-4985-95f4-4058c691b840" (UID: "421d7f0d-0d50-4985-95f4-4058c691b840"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.209068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.209037    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40ba6924-2d7d-41bc-9749-458e7a705e64-maas-api-service-ca" pod "40ba6924-2d7d-41bc-9749-458e7a705e64" (UID: "40ba6924-2d7d-41bc-9749-458e7a705e64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.209427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.209405    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/421d7f0d-0d50-4985-95f4-4058c691b840-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.209518 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.209433    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40ba6924-2d7d-41bc-9749-458e7a705e64-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.251126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.251087    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421d7f0d-0d50-4985-95f4-4058c691b840-kube-api-access-ndh64" pod "421d7f0d-0d50-4985-95f4-4058c691b840" (UID: "421d7f0d-0d50-4985-95f4-4058c691b840"). InnerVolumeSpecName "kube-api-access-ndh64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.251352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.251200    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40ba6924-2d7d-41bc-9749-458e7a705e64-kube-api-access-s5l8z" pod "40ba6924-2d7d-41bc-9749-458e7a705e64" (UID: "40ba6924-2d7d-41bc-9749-458e7a705e64"). InnerVolumeSpecName "kube-api-access-s5l8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.299765 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.299720    4291 generic.go:356] "Generic (PLEG): container finished" podID="2804875f-4c19-409b-8646-c17b8507f6f2" containerID="b2555189860516dc7587e52143238dc69e3625885b15cda15142b2b130a72732" exitCode=60
Oct 30 14:00:21.299938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.299793    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" event={"ID":"2804875f-4c19-409b-8646-c17b8507f6f2","Type":"ContainerDied","Data":"b2555189860516dc7587e52143238dc69e3625885b15cda15142b2b130a72732"}
Oct 30 14:00:21.299938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.299904    4291 scope.go:122] "RemoveContainer" containerID="41eb4f2eb12770da9b14754bd7eff922210e5bd06023c1641ae0cc3ccda6bd5e"
Oct 30 14:00:21.300498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.300469    4291 scope.go:122] "RemoveContainer" containerID="b2555189860516dc7587e52143238dc69e3625885b15cda15142b2b130a72732"
Oct 30 14:00:21.300940 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:21.300916    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56_odh-ai-gateway-infra(2804875f-4c19-409b-8646-c17b8507f6f2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" podUID="2804875f-4c19-409b-8646-c17b8507f6f2"
Oct 30 14:00:21.304014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.303984    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv" event={"ID":"421d7f0d-0d50-4985-95f4-4058c691b840","Type":"ContainerDied","Data":"df678e237379d173f3475c712ca8c757d6b84a366be6cd1aed3ded1806f7e287"}
Oct 30 14:00:21.304121 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.303994    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"
Oct 30 14:00:21.311029 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.310995    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5l8z\" (UniqueName: \"kubernetes.io/projected/40ba6924-2d7d-41bc-9749-458e7a705e64-kube-api-access-s5l8z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.311157 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.311036    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndh64\" (UniqueName: \"kubernetes.io/projected/421d7f0d-0d50-4985-95f4-4058c691b840-kube-api-access-ndh64\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.318341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.318308    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"
Oct 30 14:00:21.318513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.318479    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts" event={"ID":"02c6941b-9f8e-4ca3-84cc-7fa377906a40","Type":"ContainerDied","Data":"c32b5a43612b8e0d2ec988e03407f50d02c2969c1eef434ce1dcfb2af7be3718"}
Oct 30 14:00:21.321892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.321857    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" event={"ID":"cc3d3284-1405-4e29-91fd-fba47df039df","Type":"ContainerStarted","Data":"4c0392695e3f48a1ceb2636a619a72b95a25fb4ca335cb1dc1f1026e84ccb6fd"}
Oct 30 14:00:21.327704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.327625    4291 scope.go:122] "RemoveContainer" containerID="efc27b2089127888519f58c98aadbd49fab6b7a7a8846ef9aa6a77ae716a1f7c"
Oct 30 14:00:21.330092 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.330065    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" event={"ID":"5d440f70-f544-432c-a8f1-e38d9cb7ec68","Type":"ContainerStarted","Data":"89a5be78d0b91dca869adec495dec8d649a944ebf288b66790603e6fcf286357"}
Oct 30 14:00:21.335319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.335289    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerStarted","Data":"48d4c6e5f04b473816209367c7bb8000a5a38787debdf47087773c30a98bad70"}
Oct 30 14:00:21.338472 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.338446    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" event={"ID":"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0","Type":"ContainerStarted","Data":"6ceec67b32092e91a1d0d205a14e2742956e2e55380e776eb1da205db73e6cbe"}
Oct 30 14:00:21.341943 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.341925    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"
Oct 30 14:00:21.345943 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.341953    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx" event={"ID":"47367ecc-1223-4952-a0b8-986afe4951b9","Type":"ContainerDied","Data":"fe87221390dd6fb591e330daeea980e2fa05b4155e23bc905414ba6f61b94e7d"}
Oct 30 14:00:21.347419 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.347379    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"
Oct 30 14:00:21.347950 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.347381    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc" event={"ID":"40ba6924-2d7d-41bc-9749-458e7a705e64","Type":"ContainerDied","Data":"a77052963a1dcd269d973ec37f3e32abe2ba0ca54801a1cdc0cf4b87fe74cc66"}
Oct 30 14:00:21.368847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.364606    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" event={"ID":"f764f2cc-15c4-460c-930c-9b16c9fa82ab","Type":"ContainerStarted","Data":"b67b28caa85921ba477189bdaf3bea452317ecf5a6bf8f97587d0b8c1e6d4619"}
Oct 30 14:00:21.372189 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.372155    4291 scope.go:122] "RemoveContainer" containerID="3e1a6bcf2fd69482a6910cb0dbdf2b8ae79566e39103ac826541e1d56c4a443f"
Oct 30 14:00:21.375084 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.375054    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t" event={"ID":"077541bb-8a79-4080-8a2c-2644004dbe3d","Type":"ContainerDied","Data":"1e39aba0db888dcac7f80904d6b50708d58ab95931537d4658482b07e4a01457"}
Oct 30 14:00:21.375218 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.375196    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"
Oct 30 14:00:21.386087 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.386049    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerStarted","Data":"11f9a6b0086c7f767765115cd1cb9a446c8eeb09bdfae6429e490da8e283c9ef"}
Oct 30 14:00:21.389291 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.389173    4291 generic.go:356] "Generic (PLEG): container finished" podID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerID="21f7760558675c04328ff8c394f70e678d44237d71201aabb48177d90de7a14b" exitCode=60
Oct 30 14:00:21.389415 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.389245    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" event={"ID":"851ad965-5bcf-46ed-bdf4-1764a61b901a","Type":"ContainerDied","Data":"21f7760558675c04328ff8c394f70e678d44237d71201aabb48177d90de7a14b"}
Oct 30 14:00:21.389932 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.389915    4291 scope.go:122] "RemoveContainer" containerID="21f7760558675c04328ff8c394f70e678d44237d71201aabb48177d90de7a14b"
Oct 30 14:00:21.390244 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:21.390224    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j_odh-ai-gateway-infra(851ad965-5bcf-46ed-bdf4-1764a61b901a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a"
Oct 30 14:00:21.449486 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.449438    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"]
Oct 30 14:00:21.481328 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.481221    4291 scope.go:122] "RemoveContainer" containerID="36ace568ca786eab8c2e3164efbc3d3f2b35784b759aa6123ff55cdf2e422915"
Oct 30 14:00:21.542842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.542811    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"]
Oct 30 14:00:21.632844 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.632782    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"]
Oct 30 14:00:21.640883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.640271    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-310546-29889480-rrfpc"]
Oct 30 14:00:21.688776 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.688746    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"]
Oct 30 14:00:21.695834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.695794    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-22af91-29889480-9tz7t"]
Oct 30 14:00:21.707115 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.707089    4291 scope.go:122] "RemoveContainer" containerID="80c04f3db21066bb54e4ac8af6fa3494d8387af865a5e9eeefbcbb8e3f524a72"
Oct 30 14:00:21.716523 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.716367    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:21.726075 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.726049    4291 scope.go:122] "RemoveContainer" containerID="e72826d99e4299eb6984e3074a5f3d4fffb8bea49fae2035f05017c422c07259"
Oct 30 14:00:21.726818 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.726795    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:21.741765 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.741726    4291 scope.go:122] "RemoveContainer" containerID="df1f89e38ecd301c3dde00511098a83f153f641415ed79cc5afc0ad3cd7e6f34"
Oct 30 14:00:21.798453 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.798311    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:21.823361 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.823326    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-maas-api-service-ca\") pod \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") "
Oct 30 14:00:21.829532 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.828827    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-maas-api-service-ca" pod "d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" (UID: "d9af2f8f-927e-4948-bf5a-3bcb8afd89b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.829884 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.829862    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-kube-api-access-kkvhz\" (UniqueName: \"kubernetes.io/projected/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-kube-api-access-kkvhz\") pod \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\" (UID: \"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0\") "
Oct 30 14:00:21.830521 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.830501    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f297ee2-5036-4ac9-8839-c80d2eb293e5-kube-api-access-4xxf5\" (UniqueName: \"kubernetes.io/projected/8f297ee2-5036-4ac9-8839-c80d2eb293e5-kube-api-access-4xxf5\") pod \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") "
Oct 30 14:00:21.832421 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.832401    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f297ee2-5036-4ac9-8839-c80d2eb293e5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f297ee2-5036-4ac9-8839-c80d2eb293e5-maas-api-service-ca\") pod \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\" (UID: \"8f297ee2-5036-4ac9-8839-c80d2eb293e5\") "
Oct 30 14:00:21.833025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.832921    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f297ee2-5036-4ac9-8839-c80d2eb293e5-maas-api-service-ca" pod "8f297ee2-5036-4ac9-8839-c80d2eb293e5" (UID: "8f297ee2-5036-4ac9-8839-c80d2eb293e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.835360 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.835338    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8f297ee2-5036-4ac9-8839-c80d2eb293e5-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.841783 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.841759    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.844607 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.844583    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-kube-api-access-kkvhz" pod "d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" (UID: "d9af2f8f-927e-4948-bf5a-3bcb8afd89b0"). InnerVolumeSpecName "kube-api-access-kkvhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.845211 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.845126    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:21.847238 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.847214    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:21.849067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.849046    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f297ee2-5036-4ac9-8839-c80d2eb293e5-kube-api-access-4xxf5" pod "8f297ee2-5036-4ac9-8839-c80d2eb293e5" (UID: "8f297ee2-5036-4ac9-8839-c80d2eb293e5"). InnerVolumeSpecName "kube-api-access-4xxf5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.866090 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.866062    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:21.883717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.883679    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" path="/var/lib/kubelet/pods/077541bb-8a79-4080-8a2c-2644004dbe3d/volumes"
Oct 30 14:00:21.884311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.884294    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" path="/var/lib/kubelet/pods/40ba6924-2d7d-41bc-9749-458e7a705e64/volumes"
Oct 30 14:00:21.942376 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.942342    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"]
Oct 30 14:00:21.942804 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.942752    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-kube-api-access-2pzmg\" (UniqueName: \"kubernetes.io/projected/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-kube-api-access-2pzmg\") pod \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") "
Oct 30 14:00:21.942862 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.942803    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/addb6a6e-6673-4881-b84c-fa9561235a47-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/addb6a6e-6673-4881-b84c-fa9561235a47-maas-api-service-ca\") pod \"addb6a6e-6673-4881-b84c-fa9561235a47\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") "
Oct 30 14:00:21.942925 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.942908    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-maas-api-service-ca\") pod \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\" (UID: \"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6\") "
Oct 30 14:00:21.943012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.942994    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0e7264f-66cf-45f6-91db-4126497c2ae6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7264f-66cf-45f6-91db-4126497c2ae6-maas-api-service-ca\") pod \"d0e7264f-66cf-45f6-91db-4126497c2ae6\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") "
Oct 30 14:00:21.943150 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943132    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e43b7f3d-c901-4f59-a039-04c48e60d984-kube-api-access-8cgvk\" (UniqueName: \"kubernetes.io/projected/e43b7f3d-c901-4f59-a039-04c48e60d984-kube-api-access-8cgvk\") pod \"e43b7f3d-c901-4f59-a039-04c48e60d984\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") "
Oct 30 14:00:21.943239 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943171    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/addb6a6e-6673-4881-b84c-fa9561235a47-kube-api-access-76cwp\" (UniqueName: \"kubernetes.io/projected/addb6a6e-6673-4881-b84c-fa9561235a47-kube-api-access-76cwp\") pod \"addb6a6e-6673-4881-b84c-fa9561235a47\" (UID: \"addb6a6e-6673-4881-b84c-fa9561235a47\") "
Oct 30 14:00:21.943297 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943253    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0e7264f-66cf-45f6-91db-4126497c2ae6-kube-api-access-ct7z9\" (UniqueName: \"kubernetes.io/projected/d0e7264f-66cf-45f6-91db-4126497c2ae6-kube-api-access-ct7z9\") pod \"d0e7264f-66cf-45f6-91db-4126497c2ae6\" (UID: \"d0e7264f-66cf-45f6-91db-4126497c2ae6\") "
Oct 30 14:00:21.943359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943291    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/addb6a6e-6673-4881-b84c-fa9561235a47-maas-api-service-ca" pod "addb6a6e-6673-4881-b84c-fa9561235a47" (UID: "addb6a6e-6673-4881-b84c-fa9561235a47"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.943359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943308    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-maas-api-service-ca" pod "b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" (UID: "b4fb266d-2bcf-4120-b82e-1f8ebefee7a6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.943480 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943363    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e7264f-66cf-45f6-91db-4126497c2ae6-maas-api-service-ca" pod "d0e7264f-66cf-45f6-91db-4126497c2ae6" (UID: "d0e7264f-66cf-45f6-91db-4126497c2ae6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.943480 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943441    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e43b7f3d-c901-4f59-a039-04c48e60d984-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b7f3d-c901-4f59-a039-04c48e60d984-maas-api-service-ca\") pod \"e43b7f3d-c901-4f59-a039-04c48e60d984\" (UID: \"e43b7f3d-c901-4f59-a039-04c48e60d984\") "
Oct 30 14:00:21.944622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.943829    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e43b7f3d-c901-4f59-a039-04c48e60d984-maas-api-service-ca" pod "e43b7f3d-c901-4f59-a039-04c48e60d984" (UID: "e43b7f3d-c901-4f59-a039-04c48e60d984"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:21.944622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.944182    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xxf5\" (UniqueName: \"kubernetes.io/projected/8f297ee2-5036-4ac9-8839-c80d2eb293e5-kube-api-access-4xxf5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.944622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.944196    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e43b7f3d-c901-4f59-a039-04c48e60d984-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.944622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.944206    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/addb6a6e-6673-4881-b84c-fa9561235a47-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.944622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.944215    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkvhz\" (UniqueName: \"kubernetes.io/projected/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0-kube-api-access-kkvhz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.944622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.944223    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.944622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.944232    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0e7264f-66cf-45f6-91db-4126497c2ae6-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:21.945710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.945619    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-kube-api-access-2pzmg" pod "b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" (UID: "b4fb266d-2bcf-4120-b82e-1f8ebefee7a6"). InnerVolumeSpecName "kube-api-access-2pzmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.945869 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.945844    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e7264f-66cf-45f6-91db-4126497c2ae6-kube-api-access-ct7z9" pod "d0e7264f-66cf-45f6-91db-4126497c2ae6" (UID: "d0e7264f-66cf-45f6-91db-4126497c2ae6"). InnerVolumeSpecName "kube-api-access-ct7z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.945963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.945887    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/addb6a6e-6673-4881-b84c-fa9561235a47-kube-api-access-76cwp" pod "addb6a6e-6673-4881-b84c-fa9561235a47" (UID: "addb6a6e-6673-4881-b84c-fa9561235a47"). InnerVolumeSpecName "kube-api-access-76cwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.946753 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.946732    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e43b7f3d-c901-4f59-a039-04c48e60d984-kube-api-access-8cgvk" pod "e43b7f3d-c901-4f59-a039-04c48e60d984" (UID: "e43b7f3d-c901-4f59-a039-04c48e60d984"). InnerVolumeSpecName "kube-api-access-8cgvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:21.949826 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.949800    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-09014b-29889480-hrhzx"]
Oct 30 14:00:21.976421 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.976331    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"]
Oct 30 14:00:21.981193 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:21.981160    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-08c365-29889480-5cjzv"]
Oct 30 14:00:22.047261 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.047216    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cgvk\" (UniqueName: \"kubernetes.io/projected/e43b7f3d-c901-4f59-a039-04c48e60d984-kube-api-access-8cgvk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.047261 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.047260    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76cwp\" (UniqueName: \"kubernetes.io/projected/addb6a6e-6673-4881-b84c-fa9561235a47-kube-api-access-76cwp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.047507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.047274    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct7z9\" (UniqueName: \"kubernetes.io/projected/d0e7264f-66cf-45f6-91db-4126497c2ae6-kube-api-access-ct7z9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.047507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.047291    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pzmg\" (UniqueName: \"kubernetes.io/projected/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6-kube-api-access-2pzmg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.062180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.061148    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"]
Oct 30 14:00:22.076865 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.076832    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-844856-29889480-hgmts"]
Oct 30 14:00:22.377983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.377923    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"]
Oct 30 14:00:22.408193 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.408155    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx" event={"ID":"8f297ee2-5036-4ac9-8839-c80d2eb293e5","Type":"ContainerDied","Data":"80ffff421692a56aea710de3e5afdd430360216c5fd82253f263f87457c0c90c"}
Oct 30 14:00:22.408379 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.408170    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"
Oct 30 14:00:22.408379 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.408217    4291 scope.go:122] "RemoveContainer" containerID="ce7fd100ecef92ac712d362e997a58eeb4a9428eaf75b8a11a37a3e78b294634"
Oct 30 14:00:22.411792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.411762    4291 generic.go:356] "Generic (PLEG): container finished" podID="cc3d3284-1405-4e29-91fd-fba47df039df" containerID="4c0392695e3f48a1ceb2636a619a72b95a25fb4ca335cb1dc1f1026e84ccb6fd" exitCode=60
Oct 30 14:00:22.411953 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.411818    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" event={"ID":"cc3d3284-1405-4e29-91fd-fba47df039df","Type":"ContainerDied","Data":"4c0392695e3f48a1ceb2636a619a72b95a25fb4ca335cb1dc1f1026e84ccb6fd"}
Oct 30 14:00:22.412384 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.412360    4291 scope.go:122] "RemoveContainer" containerID="4c0392695e3f48a1ceb2636a619a72b95a25fb4ca335cb1dc1f1026e84ccb6fd"
Oct 30 14:00:22.412859 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.412827    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x_odh-ai-gateway-infra(cc3d3284-1405-4e29-91fd-fba47df039df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" podUID="cc3d3284-1405-4e29-91fd-fba47df039df"
Oct 30 14:00:22.413460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.413427    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg" event={"ID":"e43b7f3d-c901-4f59-a039-04c48e60d984","Type":"ContainerDied","Data":"90ee5372eb01592a399bdf52e38b0105f0f1120dba143bca152f0f454902c21c"}
Oct 30 14:00:22.413547 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.413436    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"
Oct 30 14:00:22.415514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.415494    4291 generic.go:356] "Generic (PLEG): container finished" podID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerID="89a5be78d0b91dca869adec495dec8d649a944ebf288b66790603e6fcf286357" exitCode=60
Oct 30 14:00:22.415616 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.415572    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" event={"ID":"5d440f70-f544-432c-a8f1-e38d9cb7ec68","Type":"ContainerDied","Data":"89a5be78d0b91dca869adec495dec8d649a944ebf288b66790603e6fcf286357"}
Oct 30 14:00:22.416006 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.415989    4291 scope.go:122] "RemoveContainer" containerID="89a5be78d0b91dca869adec495dec8d649a944ebf288b66790603e6fcf286357"
Oct 30 14:00:22.416281 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.416258    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz_odh-ai-gateway-infra(5d440f70-f544-432c-a8f1-e38d9cb7ec68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68"
Oct 30 14:00:22.417175 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.417153    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"
Oct 30 14:00:22.417285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.417153    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4" event={"ID":"addb6a6e-6673-4881-b84c-fa9561235a47","Type":"ContainerDied","Data":"042d0f693d1b060f5f40fc6a57a11a0399f65c268f124d3bed24e285c71c5929"}
Oct 30 14:00:22.419491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.419426    4291 generic.go:356] "Generic (PLEG): container finished" podID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerID="48d4c6e5f04b473816209367c7bb8000a5a38787debdf47087773c30a98bad70" exitCode=60
Oct 30 14:00:22.419600 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.419504    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerDied","Data":"48d4c6e5f04b473816209367c7bb8000a5a38787debdf47087773c30a98bad70"}
Oct 30 14:00:22.421791 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.421606    4291 scope.go:122] "RemoveContainer" containerID="48d4c6e5f04b473816209367c7bb8000a5a38787debdf47087773c30a98bad70"
Oct 30 14:00:22.421900 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.421876    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6_odh-ai-gateway-infra(398137cb-99db-4931-bca0-f6e1b1b931a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7"
Oct 30 14:00:22.422814 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.422793    4291 scope.go:122] "RemoveContainer" containerID="5b76972c6606a17ce98a8814eacc02f3c64eab2978ab061f7c18c870f0a8986b"
Oct 30 14:00:22.425043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.425021    4291 generic.go:356] "Generic (PLEG): container finished" podID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerID="6ceec67b32092e91a1d0d205a14e2742956e2e55380e776eb1da205db73e6cbe" exitCode=60
Oct 30 14:00:22.425218 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.425059    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" event={"ID":"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0","Type":"ContainerDied","Data":"6ceec67b32092e91a1d0d205a14e2742956e2e55380e776eb1da205db73e6cbe"}
Oct 30 14:00:22.425717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.425696    4291 scope.go:122] "RemoveContainer" containerID="6ceec67b32092e91a1d0d205a14e2742956e2e55380e776eb1da205db73e6cbe"
Oct 30 14:00:22.426185 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.426157    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5_odh-ai-gateway-infra(bf93e80a-eb0c-447c-a9c6-a37be5c77ab0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0"
Oct 30 14:00:22.430445 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.430424    4291 generic.go:356] "Generic (PLEG): container finished" podID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerID="b67b28caa85921ba477189bdaf3bea452317ecf5a6bf8f97587d0b8c1e6d4619" exitCode=60
Oct 30 14:00:22.430542 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.430491    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" event={"ID":"f764f2cc-15c4-460c-930c-9b16c9fa82ab","Type":"ContainerDied","Data":"b67b28caa85921ba477189bdaf3bea452317ecf5a6bf8f97587d0b8c1e6d4619"}
Oct 30 14:00:22.430901 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.430883    4291 scope.go:122] "RemoveContainer" containerID="b67b28caa85921ba477189bdaf3bea452317ecf5a6bf8f97587d0b8c1e6d4619"
Oct 30 14:00:22.431161 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.431142    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj_odh-ai-gateway-infra(f764f2cc-15c4-460c-930c-9b16c9fa82ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab"
Oct 30 14:00:22.433651 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.433615    4291 generic.go:356] "Generic (PLEG): container finished" podID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerID="11f9a6b0086c7f767765115cd1cb9a446c8eeb09bdfae6429e490da8e283c9ef" exitCode=60
Oct 30 14:00:22.433739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.433673    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerDied","Data":"11f9a6b0086c7f767765115cd1cb9a446c8eeb09bdfae6429e490da8e283c9ef"}
Oct 30 14:00:22.434111 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.434078    4291 scope.go:122] "RemoveContainer" containerID="11f9a6b0086c7f767765115cd1cb9a446c8eeb09bdfae6429e490da8e283c9ef"
Oct 30 14:00:22.434410 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.434383    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92_odh-ai-gateway-infra(f5fb8d13-b0d8-439d-b7b2-9227b69604e7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7"
Oct 30 14:00:22.435878 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.435855    4291 generic.go:356] "Generic (PLEG): container finished" podID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerID="806b4f7e448a3b0abdcb1fca6d1e8c7e54b27accbe8ed6cfe7180ad7664ee851" exitCode=60
Oct 30 14:00:22.435981 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.435877    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" event={"ID":"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b","Type":"ContainerDied","Data":"806b4f7e448a3b0abdcb1fca6d1e8c7e54b27accbe8ed6cfe7180ad7664ee851"}
Oct 30 14:00:22.436249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.436231    4291 scope.go:122] "RemoveContainer" containerID="806b4f7e448a3b0abdcb1fca6d1e8c7e54b27accbe8ed6cfe7180ad7664ee851"
Oct 30 14:00:22.436511 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.436489    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr_odh-ai-gateway-infra(4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b"
Oct 30 14:00:22.437931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.437912    4291 generic.go:356] "Generic (PLEG): container finished" podID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerID="5fd971ebeb748ac6edbe7cfff4375ad3b33e2fa40c66face1b2b7d4b0dd61df5" exitCode=60
Oct 30 14:00:22.438041 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.437985    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" event={"ID":"9a630f92-dc84-454b-8f37-c80cdcfe40b1","Type":"ContainerDied","Data":"5fd971ebeb748ac6edbe7cfff4375ad3b33e2fa40c66face1b2b7d4b0dd61df5"}
Oct 30 14:00:22.438385 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.438365    4291 scope.go:122] "RemoveContainer" containerID="5fd971ebeb748ac6edbe7cfff4375ad3b33e2fa40c66face1b2b7d4b0dd61df5"
Oct 30 14:00:22.438711 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:22.438684    4291 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889480-hkr66_odh-ai-gateway-infra(9a630f92-dc84-454b-8f37-c80cdcfe40b1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1"
Oct 30 14:00:22.440871 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.440855    4291 scope.go:122] "RemoveContainer" containerID="c5cf0136bb38683bb3a29d9b444d4f9a9d3b6701ec9d7d65c72d3a27164b3544"
Oct 30 14:00:22.441056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.441038    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"
Oct 30 14:00:22.441119 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.441051    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd" event={"ID":"d0e7264f-66cf-45f6-91db-4126497c2ae6","Type":"ContainerDied","Data":"612e75c27c996cbedf5e7ba8f30a116da9e45023fdeced9c6305d6246b94fffb"}
Oct 30 14:00:22.442542 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.442518    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"
Oct 30 14:00:22.442662 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.442541    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp" event={"ID":"b4fb266d-2bcf-4120-b82e-1f8ebefee7a6","Type":"ContainerDied","Data":"c886bf6a072d2b790d6b0e1f1a90488c6637f52bc5e202b3346a29c47eebb84a"}
Oct 30 14:00:22.446198 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.446174    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r" event={"ID":"d9af2f8f-927e-4948-bf5a-3bcb8afd89b0","Type":"ContainerDied","Data":"0267f1defe3cc65bfda8bee020fb68a1e37c4f34d2039939a300e5ab2ffcefa9"}
Oct 30 14:00:22.446291 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.446221    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"
Oct 30 14:00:22.455723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.455702    4291 scope.go:122] "RemoveContainer" containerID="7a0bdb437ee0ba39c54a1a20fda64a272721f88c51fe00710590404a4d8f995a"
Oct 30 14:00:22.472652 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.472607    4291 scope.go:122] "RemoveContainer" containerID="4ff9e2f09e817fcf44ef473194bf3fd394662bf628a7b8d9c7cc16a45b9840cc"
Oct 30 14:00:22.484814 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.484791    4291 scope.go:122] "RemoveContainer" containerID="8dc9b0e26d75e6a7f2191b7990f31c8d31f55dd60f69846edaa86cffb9bba98f"
Oct 30 14:00:22.503839 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.503805    4291 scope.go:122] "RemoveContainer" containerID="6f6e5890c15e39c9c568b2dde795cf79e9b6d3569235cc2e02a69f7d74ca7c11"
Oct 30 14:00:22.504750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.504723    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"]
Oct 30 14:00:22.533402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.533375    4291 scope.go:122] "RemoveContainer" containerID="94e30ae236f35cc889435021a1520a1333b48ce189269a0674bdb1b0d59cfe21"
Oct 30 14:00:22.554525 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.554491    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"]
Oct 30 14:00:22.556302 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.556275    4291 scope.go:122] "RemoveContainer" containerID="4cb8e0cac053d24d9c5331f759978129efbb774d0dcc78692923dbdeaed61673"
Oct 30 14:00:22.577837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.577801    4291 scope.go:122] "RemoveContainer" containerID="91db4841d357c894bb50166600f33ac40fb467bf47ac53a66cb04056434634a9"
Oct 30 14:00:22.596024 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.594932    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"]
Oct 30 14:00:22.601803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.601215    4291 scope.go:122] "RemoveContainer" containerID="d0eb86cf395d8ba783672111f87e1319dd956866585898015355400db5248e2f"
Oct 30 14:00:22.601932 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.601843    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-cab6c5-29889480-pjkgx"]
Oct 30 14:00:22.634589 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.634532    4291 scope.go:122] "RemoveContainer" containerID="1b7eaafa7678001e2024b30be03e7503963dffe7fe8bd0d3f96d0843343adb73"
Oct 30 14:00:22.635135 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.635109    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:22.648316 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.648288    4291 scope.go:122] "RemoveContainer" containerID="1bf1e67a930cd0aea456f6227cb25bb54684db3b301fc75921367fa4d9e5b8b0"
Oct 30 14:00:22.660725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.657025    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2804875f-4c19-409b-8646-c17b8507f6f2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2804875f-4c19-409b-8646-c17b8507f6f2-maas-api-service-ca\") pod \"2804875f-4c19-409b-8646-c17b8507f6f2\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") "
Oct 30 14:00:22.660725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.657084    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2804875f-4c19-409b-8646-c17b8507f6f2-kube-api-access-tqlzs\" (UniqueName: \"kubernetes.io/projected/2804875f-4c19-409b-8646-c17b8507f6f2-kube-api-access-tqlzs\") pod \"2804875f-4c19-409b-8646-c17b8507f6f2\" (UID: \"2804875f-4c19-409b-8646-c17b8507f6f2\") "
Oct 30 14:00:22.660725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.657357    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2804875f-4c19-409b-8646-c17b8507f6f2-maas-api-service-ca" pod "2804875f-4c19-409b-8646-c17b8507f6f2" (UID: "2804875f-4c19-409b-8646-c17b8507f6f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:22.660725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.657686    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2804875f-4c19-409b-8646-c17b8507f6f2-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.660725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.659209    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2804875f-4c19-409b-8646-c17b8507f6f2-kube-api-access-tqlzs" pod "2804875f-4c19-409b-8646-c17b8507f6f2" (UID: "2804875f-4c19-409b-8646-c17b8507f6f2"). InnerVolumeSpecName "kube-api-access-tqlzs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:22.665475 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.665457    4291 scope.go:122] "RemoveContainer" containerID="86a75d45dfb4587381380cec3f5e72d264161eb4ce3b89bf76672fd6377791e7"
Oct 30 14:00:22.669160 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.669138    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"]
Oct 30 14:00:22.681780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.681758    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:22.728514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.728478    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"]
Oct 30 14:00:22.732918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.732894    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-926ff0-29889480-8rlxd"]
Oct 30 14:00:22.742864 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.742838    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"]
Oct 30 14:00:22.758435 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.758406    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851ad965-5bcf-46ed-bdf4-1764a61b901a-kube-api-access-d4ks7\" (UniqueName: \"kubernetes.io/projected/851ad965-5bcf-46ed-bdf4-1764a61b901a-kube-api-access-d4ks7\") pod \"851ad965-5bcf-46ed-bdf4-1764a61b901a\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") "
Oct 30 14:00:22.758587 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.758526    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851ad965-5bcf-46ed-bdf4-1764a61b901a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851ad965-5bcf-46ed-bdf4-1764a61b901a-maas-api-service-ca\") pod \"851ad965-5bcf-46ed-bdf4-1764a61b901a\" (UID: \"851ad965-5bcf-46ed-bdf4-1764a61b901a\") "
Oct 30 14:00:22.758902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.758879    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851ad965-5bcf-46ed-bdf4-1764a61b901a-maas-api-service-ca" pod "851ad965-5bcf-46ed-bdf4-1764a61b901a" (UID: "851ad965-5bcf-46ed-bdf4-1764a61b901a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:22.759004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.758991    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851ad965-5bcf-46ed-bdf4-1764a61b901a-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.759043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.759006    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqlzs\" (UniqueName: \"kubernetes.io/projected/2804875f-4c19-409b-8646-c17b8507f6f2-kube-api-access-tqlzs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.760744 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.760717    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851ad965-5bcf-46ed-bdf4-1764a61b901a-kube-api-access-d4ks7" pod "851ad965-5bcf-46ed-bdf4-1764a61b901a" (UID: "851ad965-5bcf-46ed-bdf4-1764a61b901a"). InnerVolumeSpecName "kube-api-access-d4ks7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:22.764996 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.764972    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"]
Oct 30 14:00:22.774869 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.774844    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-167965-29889480-x54mg"]
Oct 30 14:00:22.809028 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.808996    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"]
Oct 30 14:00:22.822914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.822886    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-bd2fed-29889480-fmf9r"]
Oct 30 14:00:22.837877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.837833    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"]
Oct 30 14:00:22.839035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839013    4291 cpu_manager.go:414] "containerMap: removing container" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:22.839079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839039    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:22.839079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839054    4291 cpu_manager.go:414] "containerMap: removing container" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:22.839079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839062    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:22.839079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839076    4291 cpu_manager.go:414] "containerMap: removing container" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839083    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839096    4291 cpu_manager.go:414] "containerMap: removing container" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839104    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839121    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839128    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839138    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839145    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839158    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839165    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839175    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839182    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839197    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839204    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839224    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerName="cleanup"
Oct 30 14:00:22.839231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839232    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839245    4291 cpu_manager.go:414] "containerMap: removing container" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839252    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839263    4291 cpu_manager.go:414] "containerMap: removing container" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839269    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839284    4291 cpu_manager.go:414] "containerMap: removing container" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839291    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839301    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839308    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839317    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839324    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839340    4291 cpu_manager.go:414] "containerMap: removing container" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839346    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839363    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839370    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839378    4291 cpu_manager.go:414] "containerMap: removing container" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839385    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839397    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839404    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839421    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839428    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839438    4291 cpu_manager.go:414] "containerMap: removing container" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839445    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839455    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839462    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839477    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839486    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839498    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839505    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839516    4291 cpu_manager.go:414] "containerMap: removing container" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839523    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839541    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839548    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839559    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839566    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839585    4291 cpu_manager.go:414] "containerMap: removing container" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839592    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839603    4291 cpu_manager.go:414] "containerMap: removing container" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839609    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839622    4291 cpu_manager.go:414] "containerMap: removing container" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839628    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839684    4291 cpu_manager.go:414] "containerMap: removing container" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839691    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839706    4291 cpu_manager.go:414] "containerMap: removing container" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839713    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839729    4291 cpu_manager.go:414] "containerMap: removing container" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:22.839740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839735    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839947    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839959    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839967    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839977    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839986    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.839995    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840002    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840013    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840021    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840030    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840037    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840047    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840060    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840069    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840078    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840085    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840094    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840103    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840113    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b5fc286-5675-4c60-bb29-9b0bf79bd701" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840124    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840133    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840141    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840151    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840160    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840169    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840179    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840188    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="077541bb-8a79-4080-8a2c-2644004dbe3d" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840199    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840207    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840220    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840227    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840236    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840246    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:22.841057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.840255    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:22.844730 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.844709    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"]
Oct 30 14:00:22.844826 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.844809    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.861078 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.861041    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7w2n\" (UniqueName: \"kubernetes.io/projected/45f0dda2-ba36-42a4-bc8b-e67f329c007f-kube-api-access-c7w2n\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.861267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.861249    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.861380 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.861348    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.862060 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.861529    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4ks7\" (UniqueName: \"kubernetes.io/projected/851ad965-5bcf-46ed-bdf4-1764a61b901a-kube-api-access-d4ks7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:22.862060 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.862023    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:00:22.863803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.863771    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"]
Oct 30 14:00:22.879382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.879349    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"]
Oct 30 14:00:22.879547 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.879393    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-e3eb98-29889480-qtcd4"]
Oct 30 14:00:22.897933 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.897840    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"]
Oct 30 14:00:22.915028 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.915000    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"]
Oct 30 14:00:22.923485 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.923454    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"]
Oct 30 14:00:22.930679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.930644    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-d724c8-29889480-8gzfp"]
Oct 30 14:00:22.963135 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.963091    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.963682 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.963242    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7w2n\" (UniqueName: \"kubernetes.io/projected/45f0dda2-ba36-42a4-bc8b-e67f329c007f-kube-api-access-c7w2n\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.963682 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.963397    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.964073 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.964052    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.965578 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.965558    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:22.982518 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:22.982489    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7w2n\" (UniqueName: \"kubernetes.io/projected/45f0dda2-ba36-42a4-bc8b-e67f329c007f-kube-api-access-c7w2n\") pod \"maas-api-revoke-sub-54c67a3e-qdbpg\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:23.163801 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.163720    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:23.457917 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.457832    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"
Oct 30 14:00:23.458080 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.457831    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56" event={"ID":"2804875f-4c19-409b-8646-c17b8507f6f2","Type":"ContainerDied","Data":"dad9960baed21831b000e11b66d7137dff3f8d65910941ec87ae6d7e1613567b"}
Oct 30 14:00:23.458080 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.458002    4291 scope.go:122] "RemoveContainer" containerID="b2555189860516dc7587e52143238dc69e3625885b15cda15142b2b130a72732"
Oct 30 14:00:23.495444 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:23.495192    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f0dda2_ba36_42a4_bc8b_e67f329c007f.slice/crio-33a90f14342c6455ee158801cd6e763ce59b68b6ea040e99fd15aa5a97bcbc58 WatchSource:0}: Error finding container 33a90f14342c6455ee158801cd6e763ce59b68b6ea040e99fd15aa5a97bcbc58: Status 404 returned error can't find the container with id 33a90f14342c6455ee158801cd6e763ce59b68b6ea040e99fd15aa5a97bcbc58
Oct 30 14:00:23.504754 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.499477    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j" event={"ID":"851ad965-5bcf-46ed-bdf4-1764a61b901a","Type":"ContainerDied","Data":"40db06c64ff24b17966e1f2239bbbe97e0a96b1f564097f2a10f71f839b46f89"}
Oct 30 14:00:23.504754 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.499549    4291 scope.go:122] "RemoveContainer" containerID="21f7760558675c04328ff8c394f70e678d44237d71201aabb48177d90de7a14b"
Oct 30 14:00:23.504754 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.499765    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"
Oct 30 14:00:23.514602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.511719    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"]
Oct 30 14:00:23.609887 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.609791    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"]
Oct 30 14:00:23.616206 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.616166    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c3efb2-29889480-vgw56"]
Oct 30 14:00:23.671305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.671270    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:23.686598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.686563    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"]
Oct 30 14:00:23.691520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.691491    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-f12b80-29889480-d2w9j"]
Oct 30 14:00:23.749975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.749945    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:23.783553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.779791    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc3d3284-1405-4e29-91fd-fba47df039df-kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/cc3d3284-1405-4e29-91fd-fba47df039df-kube-api-access-2t4vz\") pod \"cc3d3284-1405-4e29-91fd-fba47df039df\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") "
Oct 30 14:00:23.783553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.780500    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc3d3284-1405-4e29-91fd-fba47df039df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d3284-1405-4e29-91fd-fba47df039df-maas-api-service-ca\") pod \"cc3d3284-1405-4e29-91fd-fba47df039df\" (UID: \"cc3d3284-1405-4e29-91fd-fba47df039df\") "
Oct 30 14:00:23.783553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.781157    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc3d3284-1405-4e29-91fd-fba47df039df-maas-api-service-ca" pod "cc3d3284-1405-4e29-91fd-fba47df039df" (UID: "cc3d3284-1405-4e29-91fd-fba47df039df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:23.783553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.782116    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc3d3284-1405-4e29-91fd-fba47df039df-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:23.783553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.783151    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc3d3284-1405-4e29-91fd-fba47df039df-kube-api-access-2t4vz" pod "cc3d3284-1405-4e29-91fd-fba47df039df" (UID: "cc3d3284-1405-4e29-91fd-fba47df039df"). InnerVolumeSpecName "kube-api-access-2t4vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:23.828892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.828866    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:23.883655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.883154    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d440f70-f544-432c-a8f1-e38d9cb7ec68-kube-api-access-ljw5j\" (UniqueName: \"kubernetes.io/projected/5d440f70-f544-432c-a8f1-e38d9cb7ec68-kube-api-access-ljw5j\") pod \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") "
Oct 30 14:00:23.883655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.883359    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d440f70-f544-432c-a8f1-e38d9cb7ec68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d440f70-f544-432c-a8f1-e38d9cb7ec68-maas-api-service-ca\") pod \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\" (UID: \"5d440f70-f544-432c-a8f1-e38d9cb7ec68\") "
Oct 30 14:00:23.883957 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.883903    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2t4vz\" (UniqueName: \"kubernetes.io/projected/cc3d3284-1405-4e29-91fd-fba47df039df-kube-api-access-2t4vz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:23.884218 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.884196    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d440f70-f544-432c-a8f1-e38d9cb7ec68-maas-api-service-ca" pod "5d440f70-f544-432c-a8f1-e38d9cb7ec68" (UID: "5d440f70-f544-432c-a8f1-e38d9cb7ec68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:23.887110 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.887081    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d440f70-f544-432c-a8f1-e38d9cb7ec68-kube-api-access-ljw5j" pod "5d440f70-f544-432c-a8f1-e38d9cb7ec68" (UID: "5d440f70-f544-432c-a8f1-e38d9cb7ec68"). InnerVolumeSpecName "kube-api-access-ljw5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:23.904932 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.904895    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" path="/var/lib/kubelet/pods/02c6941b-9f8e-4ca3-84cc-7fa377906a40/volumes"
Oct 30 14:00:23.906080 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.906055    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" path="/var/lib/kubelet/pods/2804875f-4c19-409b-8646-c17b8507f6f2/volumes"
Oct 30 14:00:23.906874 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.906851    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" path="/var/lib/kubelet/pods/421d7f0d-0d50-4985-95f4-4058c691b840/volumes"
Oct 30 14:00:23.908049 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.908028    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" path="/var/lib/kubelet/pods/47367ecc-1223-4952-a0b8-986afe4951b9/volumes"
Oct 30 14:00:23.908831 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.908811    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" path="/var/lib/kubelet/pods/851ad965-5bcf-46ed-bdf4-1764a61b901a/volumes"
Oct 30 14:00:23.909539 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.909520    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" path="/var/lib/kubelet/pods/8f297ee2-5036-4ac9-8839-c80d2eb293e5/volumes"
Oct 30 14:00:23.910720 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.910625    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" path="/var/lib/kubelet/pods/addb6a6e-6673-4881-b84c-fa9561235a47/volumes"
Oct 30 14:00:23.911387 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.911368    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" path="/var/lib/kubelet/pods/b4fb266d-2bcf-4120-b82e-1f8ebefee7a6/volumes"
Oct 30 14:00:23.925826 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.923834    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:23.928664 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.926339    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" path="/var/lib/kubelet/pods/d0e7264f-66cf-45f6-91db-4126497c2ae6/volumes"
Oct 30 14:00:23.928664 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.927042    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9af2f8f-927e-4948-bf5a-3bcb8afd89b0" path="/var/lib/kubelet/pods/d9af2f8f-927e-4948-bf5a-3bcb8afd89b0/volumes"
Oct 30 14:00:23.928664 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.928107    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" path="/var/lib/kubelet/pods/e43b7f3d-c901-4f59-a039-04c48e60d984/volumes"
Oct 30 14:00:23.942212 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.941545    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:23.943789 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.943766    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:23.985194 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.985165    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-maas-api-service-ca\") pod \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") "
Oct 30 14:00:23.985609 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.985305    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-kube-api-access-bvv5v\" (UniqueName: \"kubernetes.io/projected/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-kube-api-access-bvv5v\") pod \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\" (UID: \"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0\") "
Oct 30 14:00:23.986200 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.986174    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5d440f70-f544-432c-a8f1-e38d9cb7ec68-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:23.986288 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.986204    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljw5j\" (UniqueName: \"kubernetes.io/projected/5d440f70-f544-432c-a8f1-e38d9cb7ec68-kube-api-access-ljw5j\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:23.987015 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.986955    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-maas-api-service-ca" pod "bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" (UID: "bf93e80a-eb0c-447c-a9c6-a37be5c77ab0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:23.988703 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:23.988673    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-kube-api-access-bvv5v" pod "bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" (UID: "bf93e80a-eb0c-447c-a9c6-a37be5c77ab0"). InnerVolumeSpecName "kube-api-access-bvv5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:24.084425 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.084393    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"]
Oct 30 14:00:24.087512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.087474    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f764f2cc-15c4-460c-930c-9b16c9fa82ab-kube-api-access-p5l6g\" (UniqueName: \"kubernetes.io/projected/f764f2cc-15c4-460c-930c-9b16c9fa82ab-kube-api-access-p5l6g\") pod \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") "
Oct 30 14:00:24.087512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.087517    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-maas-api-service-ca\") pod \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") "
Oct 30 14:00:24.087740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.087578    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398137cb-99db-4931-bca0-f6e1b1b931a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/398137cb-99db-4931-bca0-f6e1b1b931a7-maas-api-service-ca\") pod \"398137cb-99db-4931-bca0-f6e1b1b931a7\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") "
Oct 30 14:00:24.087740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.087724    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398137cb-99db-4931-bca0-f6e1b1b931a7-kube-api-access-rmswf\" (UniqueName: \"kubernetes.io/projected/398137cb-99db-4931-bca0-f6e1b1b931a7-kube-api-access-rmswf\") pod \"398137cb-99db-4931-bca0-f6e1b1b931a7\" (UID: \"398137cb-99db-4931-bca0-f6e1b1b931a7\") "
Oct 30 14:00:24.087845 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.087767    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-kube-api-access-gqdkk\" (UniqueName: \"kubernetes.io/projected/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-kube-api-access-gqdkk\") pod \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\" (UID: \"f5fb8d13-b0d8-439d-b7b2-9227b69604e7\") "
Oct 30 14:00:24.087893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.087844    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f764f2cc-15c4-460c-930c-9b16c9fa82ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f764f2cc-15c4-460c-930c-9b16c9fa82ab-maas-api-service-ca\") pod \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\" (UID: \"f764f2cc-15c4-460c-930c-9b16c9fa82ab\") "
Oct 30 14:00:24.088003 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.087976    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398137cb-99db-4931-bca0-f6e1b1b931a7-maas-api-service-ca" pod "398137cb-99db-4931-bca0-f6e1b1b931a7" (UID: "398137cb-99db-4931-bca0-f6e1b1b931a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:24.088074 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.088003    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-maas-api-service-ca" pod "f5fb8d13-b0d8-439d-b7b2-9227b69604e7" (UID: "f5fb8d13-b0d8-439d-b7b2-9227b69604e7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:24.088300 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.088281    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f764f2cc-15c4-460c-930c-9b16c9fa82ab-maas-api-service-ca" pod "f764f2cc-15c4-460c-930c-9b16c9fa82ab" (UID: "f764f2cc-15c4-460c-930c-9b16c9fa82ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:24.088656 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.088610    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvv5v\" (UniqueName: \"kubernetes.io/projected/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-kube-api-access-bvv5v\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.088743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.088659    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.088743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.088677    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/398137cb-99db-4931-bca0-f6e1b1b931a7-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.088743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.088691    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f764f2cc-15c4-460c-930c-9b16c9fa82ab-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.088743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.088706    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.091352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.091319    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398137cb-99db-4931-bca0-f6e1b1b931a7-kube-api-access-rmswf" pod "398137cb-99db-4931-bca0-f6e1b1b931a7" (UID: "398137cb-99db-4931-bca0-f6e1b1b931a7"). InnerVolumeSpecName "kube-api-access-rmswf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:24.092247 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.092220    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-kube-api-access-gqdkk" pod "f5fb8d13-b0d8-439d-b7b2-9227b69604e7" (UID: "f5fb8d13-b0d8-439d-b7b2-9227b69604e7"). InnerVolumeSpecName "kube-api-access-gqdkk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:24.093717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.093689    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f764f2cc-15c4-460c-930c-9b16c9fa82ab-kube-api-access-p5l6g" pod "f764f2cc-15c4-460c-930c-9b16c9fa82ab" (UID: "f764f2cc-15c4-460c-930c-9b16c9fa82ab"). InnerVolumeSpecName "kube-api-access-p5l6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:24.131525 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.131475    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"]
Oct 30 14:00:24.194263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.194224    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmswf\" (UniqueName: \"kubernetes.io/projected/398137cb-99db-4931-bca0-f6e1b1b931a7-kube-api-access-rmswf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.194450 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.194264    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqdkk\" (UniqueName: \"kubernetes.io/projected/f5fb8d13-b0d8-439d-b7b2-9227b69604e7-kube-api-access-gqdkk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.194450 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.194283    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5l6g\" (UniqueName: \"kubernetes.io/projected/f764f2cc-15c4-460c-930c-9b16c9fa82ab-kube-api-access-p5l6g\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.208021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.206702    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k" podUID="e68c3640-a5a8-4641-ab24-27c7f347b239" containerName="main" containerID="cri-o://ba0b1650a741a93d394f6958c217836604a71ed29e3f023928e8cc7ead3005d3" gracePeriod=45
Oct 30 14:00:24.333805 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.333777    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:24.355480 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.355450    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:24.478659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.436262    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-kube-api-access-272tx\" (UniqueName: \"kubernetes.io/projected/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-kube-api-access-272tx\") pod \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") "
Oct 30 14:00:24.478659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.436411    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a630f92-dc84-454b-8f37-c80cdcfe40b1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a630f92-dc84-454b-8f37-c80cdcfe40b1-maas-api-service-ca\") pod \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") "
Oct 30 14:00:24.478659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.436484    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-maas-api-service-ca\") pod \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\" (UID: \"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b\") "
Oct 30 14:00:24.478659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.436600    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a630f92-dc84-454b-8f37-c80cdcfe40b1-kube-api-access-d4b5h\" (UniqueName: \"kubernetes.io/projected/9a630f92-dc84-454b-8f37-c80cdcfe40b1-kube-api-access-d4b5h\") pod \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\" (UID: \"9a630f92-dc84-454b-8f37-c80cdcfe40b1\") "
Oct 30 14:00:24.543187 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.524423    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k" podUID="e68c3640-a5a8-4641-ab24-27c7f347b239" containerName="main" probeResult="failure" output="Get \"https://10.133.1.45:8000/ready\": dial tcp 10.133.1.45:8000: connect: connection refused"
Oct 30 14:00:24.586463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.550661    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-maas-api-service-ca" pod "4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" (UID: "4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:24.586463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.560309    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.586463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.564297    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-kube-api-access-272tx" pod "4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" (UID: "4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b"). InnerVolumeSpecName "kube-api-access-272tx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:24.603693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.596830    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a630f92-dc84-454b-8f37-c80cdcfe40b1-maas-api-service-ca" pod "9a630f92-dc84-454b-8f37-c80cdcfe40b1" (UID: "9a630f92-dc84-454b-8f37-c80cdcfe40b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:24.603693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.596991    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a630f92-dc84-454b-8f37-c80cdcfe40b1-kube-api-access-d4b5h" pod "9a630f92-dc84-454b-8f37-c80cdcfe40b1" (UID: "9a630f92-dc84-454b-8f37-c80cdcfe40b1"). InnerVolumeSpecName "kube-api-access-d4b5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:24.634014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.633977    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz" event={"ID":"5d440f70-f544-432c-a8f1-e38d9cb7ec68","Type":"ContainerDied","Data":"7e9a849f5d5c72687a11605a52995584d187eaf8d293101d60f1e59018bdb176"}
Oct 30 14:00:24.634227 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.634077    4291 scope.go:122] "RemoveContainer" containerID="89a5be78d0b91dca869adec495dec8d649a944ebf288b66790603e6fcf286357"
Oct 30 14:00:24.634714 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.634684    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"
Oct 30 14:00:24.636210 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.636178    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6" event={"ID":"398137cb-99db-4931-bca0-f6e1b1b931a7","Type":"ContainerDied","Data":"c23477f38d15acc6d5c3bbf2257d688905c1ab304f7ce214be24774d3c0fe93d"}
Oct 30 14:00:24.636320 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.636300    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"
Oct 30 14:00:24.640377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.640354    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5" event={"ID":"bf93e80a-eb0c-447c-a9c6-a37be5c77ab0","Type":"ContainerDied","Data":"fc71fbd89d6e9d52e4fba30b0cb43736517c2a0c1570b77d78d896cec5d93e68"}
Oct 30 14:00:24.640550 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.640379    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"
Oct 30 14:00:24.641819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.641795    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj" event={"ID":"f764f2cc-15c4-460c-930c-9b16c9fa82ab","Type":"ContainerDied","Data":"b00f4d7cc72945bfa4712e855fd65553701928654c1982b3caf82ba027ef8126"}
Oct 30 14:00:24.641906 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.641896    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"
Oct 30 14:00:24.649747 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.649729    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"
Oct 30 14:00:24.649871 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.649769    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92" event={"ID":"f5fb8d13-b0d8-439d-b7b2-9227b69604e7","Type":"ContainerDied","Data":"6afae7eb4b38019debddb5c241ecf815138063798023673f99a6da65c7c9b57c"}
Oct 30 14:00:24.654089 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.651885    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"
Oct 30 14:00:24.654089 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.652838    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr" event={"ID":"4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b","Type":"ContainerDied","Data":"340f982a53bdd73e0ca30302d416944f015495659e14435ac0b4e50ff10900e3"}
Oct 30 14:00:24.661671 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.661626    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4b5h\" (UniqueName: \"kubernetes.io/projected/9a630f92-dc84-454b-8f37-c80cdcfe40b1-kube-api-access-d4b5h\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.661671 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.661668    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-272tx\" (UniqueName: \"kubernetes.io/projected/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b-kube-api-access-272tx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.661794 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.661679    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9a630f92-dc84-454b-8f37-c80cdcfe40b1-maas-api-service-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.662140 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.662121    4291 scope.go:122] "RemoveContainer" containerID="48d4c6e5f04b473816209367c7bb8000a5a38787debdf47087773c30a98bad70"
Oct 30 14:00:24.663108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.663094    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"
Oct 30 14:00:24.663215 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.663125    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66" event={"ID":"9a630f92-dc84-454b-8f37-c80cdcfe40b1","Type":"ContainerDied","Data":"d7db5fce4a50fcc2f33733fed6f03580bbfa7572095984d428195726f23c4961"}
Oct 30 14:00:24.670898 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.670875    4291 generic.go:356] "Generic (PLEG): container finished" podID="e68c3640-a5a8-4641-ab24-27c7f347b239" containerID="ba0b1650a741a93d394f6958c217836604a71ed29e3f023928e8cc7ead3005d3" exitCode=0
Oct 30 14:00:24.670986 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.670956    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k" event={"ID":"e68c3640-a5a8-4641-ab24-27c7f347b239","Type":"ContainerDied","Data":"ba0b1650a741a93d394f6958c217836604a71ed29e3f023928e8cc7ead3005d3"}
Oct 30 14:00:24.675147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.674510    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg" event={"ID":"45f0dda2-ba36-42a4-bc8b-e67f329c007f","Type":"ContainerStarted","Data":"cb2896653baaed9f30fabd4cf8dd853ecf6619ca49327d700350630dec68bf4d"}
Oct 30 14:00:24.675147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.674559    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg" event={"ID":"45f0dda2-ba36-42a4-bc8b-e67f329c007f","Type":"ContainerStarted","Data":"33a90f14342c6455ee158801cd6e763ce59b68b6ea040e99fd15aa5a97bcbc58"}
Oct 30 14:00:24.681372 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.681347    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x" event={"ID":"cc3d3284-1405-4e29-91fd-fba47df039df","Type":"ContainerDied","Data":"86b84a7f03d687297b7e0c071f8426d11b3ea552d74e9c802fd325a33665da22"}
Oct 30 14:00:24.681531 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.681461    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"
Oct 30 14:00:24.684428 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.684387    4291 scope.go:122] "RemoveContainer" containerID="6ceec67b32092e91a1d0d205a14e2742956e2e55380e776eb1da205db73e6cbe"
Oct 30 14:00:24.695954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.695927    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"]
Oct 30 14:00:24.701166 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.701143    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-f7ee8a-29889480-sx2cz"]
Oct 30 14:00:24.702697 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.702682    4291 scope.go:122] "RemoveContainer" containerID="b67b28caa85921ba477189bdaf3bea452317ecf5a6bf8f97587d0b8c1e6d4619"
Oct 30 14:00:24.748174 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.748140    4291 scope.go:122] "RemoveContainer" containerID="11f9a6b0086c7f767765115cd1cb9a446c8eeb09bdfae6429e490da8e283c9ef"
Oct 30 14:00:24.758804 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.758780    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 14:00:24.760316 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.760286    4291 scope.go:122] "RemoveContainer" containerID="806b4f7e448a3b0abdcb1fca6d1e8c7e54b27accbe8ed6cfe7180ad7664ee851"
Oct 30 14:00:24.762620 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.762594    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-model-cache\") pod \"e68c3640-a5a8-4641-ab24-27c7f347b239\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") "
Oct 30 14:00:24.762832 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.762810    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-home\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-home\") pod \"e68c3640-a5a8-4641-ab24-27c7f347b239\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") "
Oct 30 14:00:24.762932 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.762918    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-dshm\") pod \"e68c3640-a5a8-4641-ab24-27c7f347b239\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") "
Oct 30 14:00:24.762999 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.762957    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-model-cache" pod "e68c3640-a5a8-4641-ab24-27c7f347b239" (UID: "e68c3640-a5a8-4641-ab24-27c7f347b239"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:24.763075 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.763056    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e68c3640-a5a8-4641-ab24-27c7f347b239-kube-api-access-k85vz\" (UniqueName: \"kubernetes.io/projected/e68c3640-a5a8-4641-ab24-27c7f347b239-kube-api-access-k85vz\") pod \"e68c3640-a5a8-4641-ab24-27c7f347b239\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") "
Oct 30 14:00:24.763147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.763090    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-home" pod "e68c3640-a5a8-4641-ab24-27c7f347b239" (UID: "e68c3640-a5a8-4641-ab24-27c7f347b239"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:24.763212 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.763153    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-tmp-dir\") pod \"e68c3640-a5a8-4641-ab24-27c7f347b239\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") "
Oct 30 14:00:24.763271 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.763250    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e68c3640-a5a8-4641-ab24-27c7f347b239-tls-certs\" (UniqueName: \"kubernetes.io/secret/e68c3640-a5a8-4641-ab24-27c7f347b239-tls-certs\") pod \"e68c3640-a5a8-4641-ab24-27c7f347b239\" (UID: \"e68c3640-a5a8-4641-ab24-27c7f347b239\") "
Oct 30 14:00:24.763446 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.763419    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-tmp-dir" pod "e68c3640-a5a8-4641-ab24-27c7f347b239" (UID: "e68c3640-a5a8-4641-ab24-27c7f347b239"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:24.764109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.764047    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.764109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.764073    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.764109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.764088    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.765843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.765795    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e68c3640-a5a8-4641-ab24-27c7f347b239-tls-certs" pod "e68c3640-a5a8-4641-ab24-27c7f347b239" (UID: "e68c3640-a5a8-4641-ab24-27c7f347b239"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:00:24.765843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.765798    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68c3640-a5a8-4641-ab24-27c7f347b239-kube-api-access-k85vz" pod "e68c3640-a5a8-4641-ab24-27c7f347b239" (UID: "e68c3640-a5a8-4641-ab24-27c7f347b239"). InnerVolumeSpecName "kube-api-access-k85vz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:24.765971 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.765848    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-dshm" pod "e68c3640-a5a8-4641-ab24-27c7f347b239" (UID: "e68c3640-a5a8-4641-ab24-27c7f347b239"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:00:24.772556 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.772534    4291 scope.go:122] "RemoveContainer" containerID="5fd971ebeb748ac6edbe7cfff4375ad3b33e2fa40c66face1b2b7d4b0dd61df5"
Oct 30 14:00:24.773065 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.773045    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"]
Oct 30 14:00:24.777182 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.777160    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-552cc3-29889480-jnsg6"]
Oct 30 14:00:24.782582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.782563    4291 scope.go:122] "RemoveContainer" containerID="4c0392695e3f48a1ceb2636a619a72b95a25fb4ca335cb1dc1f1026e84ccb6fd"
Oct 30 14:00:24.810067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.810001    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"]
Oct 30 14:00:24.814098 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.814072    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-cad2c6-29889480-9kv9x"]
Oct 30 14:00:24.848515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.848460    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg" podStartSLOduration=2.848446532 podStartE2EDuration="2.848446532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:24.846831295 +0000 UTC m=+3435.107069540" watchObservedRunningTime="2026-10-30 14:00:24.848446532 +0000 UTC m=+3435.108684765"
Oct 30 14:00:24.866004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.865973    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e68c3640-a5a8-4641-ab24-27c7f347b239-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.866004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.866000    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k85vz\" (UniqueName: \"kubernetes.io/projected/e68c3640-a5a8-4641-ab24-27c7f347b239-kube-api-access-k85vz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.866004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.866011    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e68c3640-a5a8-4641-ab24-27c7f347b239-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:24.890307 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.890273    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"]
Oct 30 14:00:24.895486 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.895457    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-a75d02-29889480-lk4xj"]
Oct 30 14:00:24.930477 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.930443    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"]
Oct 30 14:00:24.934517 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.934491    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-74589c-29889480-vzcc5"]
Oct 30 14:00:24.968916 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.968880    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"]
Oct 30 14:00:24.972615 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.972590    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-319179-29889480-77f92"]
Oct 30 14:00:24.973369 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.973338    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz" podUID="ade31a0f-9282-4bde-86a9-e17e7b41f05b" containerName="payload-processing" containerID="cri-o://34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e" gracePeriod=25
Oct 30 14:00:24.995845 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:24.995805    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" podUID="6656cb26-447c-45e6-96c7-ab21cd9d6926" containerName="payload-pre-processing" containerID="cri-o://ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6" gracePeriod=25
Oct 30 14:00:25.049964 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.049923    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"]
Oct 30 14:00:25.058677 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.058627    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889480-hkr66"]
Oct 30 14:00:25.094609 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.094579    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"]
Oct 30 14:00:25.106330 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.106301    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-ca0243-29889480-xvccr"]
Oct 30 14:00:25.263377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.263347    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 14:00:25.270618 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.270589    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ade31a0f-9282-4bde-86a9-e17e7b41f05b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade31a0f-9282-4bde-86a9-e17e7b41f05b-plugins-config-volume\") pod \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") "
Oct 30 14:00:25.270783 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.270764    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ade31a0f-9282-4bde-86a9-e17e7b41f05b-kube-api-access-2lrgd\" (UniqueName: \"kubernetes.io/projected/ade31a0f-9282-4bde-86a9-e17e7b41f05b-kube-api-access-2lrgd\") pod \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\" (UID: \"ade31a0f-9282-4bde-86a9-e17e7b41f05b\") "
Oct 30 14:00:25.270976 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.270952    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ade31a0f-9282-4bde-86a9-e17e7b41f05b-plugins-config-volume" pod "ade31a0f-9282-4bde-86a9-e17e7b41f05b" (UID: "ade31a0f-9282-4bde-86a9-e17e7b41f05b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:25.271230 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.271211    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ade31a0f-9282-4bde-86a9-e17e7b41f05b-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:25.273012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.272993    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ade31a0f-9282-4bde-86a9-e17e7b41f05b-kube-api-access-2lrgd" pod "ade31a0f-9282-4bde-86a9-e17e7b41f05b" (UID: "ade31a0f-9282-4bde-86a9-e17e7b41f05b"). InnerVolumeSpecName "kube-api-access-2lrgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:25.373103 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.373065    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lrgd\" (UniqueName: \"kubernetes.io/projected/ade31a0f-9282-4bde-86a9-e17e7b41f05b-kube-api-access-2lrgd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:25.694750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.694658    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k" event={"ID":"e68c3640-a5a8-4641-ab24-27c7f347b239","Type":"ContainerDied","Data":"4d5f430be02d71c59e02831a9b2aa82c0dc504179aea4cc73c03f4b4d1992892"}
Oct 30 14:00:25.694750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.694709    4291 scope.go:122] "RemoveContainer" containerID="ba0b1650a741a93d394f6958c217836604a71ed29e3f023928e8cc7ead3005d3"
Oct 30 14:00:25.694750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.694664    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"
Oct 30 14:00:25.696454 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.696424    4291 generic.go:356] "Generic (PLEG): container finished" podID="45f0dda2-ba36-42a4-bc8b-e67f329c007f" containerID="cb2896653baaed9f30fabd4cf8dd853ecf6619ca49327d700350630dec68bf4d" exitCode=0
Oct 30 14:00:25.696576 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.696462    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg" event={"ID":"45f0dda2-ba36-42a4-bc8b-e67f329c007f","Type":"ContainerDied","Data":"cb2896653baaed9f30fabd4cf8dd853ecf6619ca49327d700350630dec68bf4d"}
Oct 30 14:00:25.702403 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.702372    4291 generic.go:356] "Generic (PLEG): container finished" podID="ade31a0f-9282-4bde-86a9-e17e7b41f05b" containerID="34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e" exitCode=0
Oct 30 14:00:25.702515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.702469    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz" event={"ID":"ade31a0f-9282-4bde-86a9-e17e7b41f05b","Type":"ContainerDied","Data":"34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e"}
Oct 30 14:00:25.702515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.702483    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"
Oct 30 14:00:25.702515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.702511    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz" event={"ID":"ade31a0f-9282-4bde-86a9-e17e7b41f05b","Type":"ContainerDied","Data":"63e1357cfa6d2649894e65ee64a7bc35ecab87a29094b43f967a834096afaec9"}
Oct 30 14:00:25.707835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.707812    4291 scope.go:122] "RemoveContainer" containerID="34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e"
Oct 30 14:00:25.719408 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.719383    4291 scope.go:122] "RemoveContainer" containerID="34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e"
Oct 30 14:00:25.719694 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:25.719673    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e\": container with ID starting with 34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e not found: ID does not exist" containerID="34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e"
Oct 30 14:00:25.719747 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.719702    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e"} err="failed to get container status \"34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e\": rpc error: code = NotFound desc = could not find container \"34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e\": container with ID starting with 34a0330df8722f863e37139ce3c75abd48bfc98ff7251c01464259470148960e not found: ID does not exist"
Oct 30 14:00:25.790181 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.790143    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"]
Oct 30 14:00:25.796731 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.796468    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-09014b/test-model-09014b-kserve-79c89fd947-r8q2k"]
Oct 30 14:00:25.883628 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.883588    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" path="/var/lib/kubelet/pods/398137cb-99db-4931-bca0-f6e1b1b931a7/volumes"
Oct 30 14:00:25.884189 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.884170    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" path="/var/lib/kubelet/pods/4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b/volumes"
Oct 30 14:00:25.884741 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.884723    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" path="/var/lib/kubelet/pods/5d440f70-f544-432c-a8f1-e38d9cb7ec68/volumes"
Oct 30 14:00:25.885598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.885573    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" path="/var/lib/kubelet/pods/9a630f92-dc84-454b-8f37-c80cdcfe40b1/volumes"
Oct 30 14:00:25.886158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.886141    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" path="/var/lib/kubelet/pods/bf93e80a-eb0c-447c-a9c6-a37be5c77ab0/volumes"
Oct 30 14:00:25.886678 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.886661    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" path="/var/lib/kubelet/pods/cc3d3284-1405-4e29-91fd-fba47df039df/volumes"
Oct 30 14:00:25.887453 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.887429    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e68c3640-a5a8-4641-ab24-27c7f347b239" path="/var/lib/kubelet/pods/e68c3640-a5a8-4641-ab24-27c7f347b239/volumes"
Oct 30 14:00:25.888071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.888048    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" path="/var/lib/kubelet/pods/f5fb8d13-b0d8-439d-b7b2-9227b69604e7/volumes"
Oct 30 14:00:25.888698 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.888676    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" path="/var/lib/kubelet/pods/f764f2cc-15c4-460c-930c-9b16c9fa82ab/volumes"
Oct 30 14:00:25.906238 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.906205    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"]
Oct 30 14:00:25.911837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:25.911801    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-08c365-85c9f96ccd-6lmhz"]
Oct 30 14:00:26.887896 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:26.887869    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:27.003579 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.003492    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45f0dda2-ba36-42a4-bc8b-e67f329c007f-kube-api-access-c7w2n\" (UniqueName: \"kubernetes.io/projected/45f0dda2-ba36-42a4-bc8b-e67f329c007f-kube-api-access-c7w2n\") pod \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") "
Oct 30 14:00:27.003866 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.003614    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-openshift-ca\") pod \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") "
Oct 30 14:00:27.003866 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.003801    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-xks-ca\") pod \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\" (UID: \"45f0dda2-ba36-42a4-bc8b-e67f329c007f\") "
Oct 30 14:00:27.004007 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.003929    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-openshift-ca" pod "45f0dda2-ba36-42a4-bc8b-e67f329c007f" (UID: "45f0dda2-ba36-42a4-bc8b-e67f329c007f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:27.004355 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.004330    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:27.005931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.005901    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-xks-ca" pod "45f0dda2-ba36-42a4-bc8b-e67f329c007f" (UID: "45f0dda2-ba36-42a4-bc8b-e67f329c007f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:00:27.006030 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.005948    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f0dda2-ba36-42a4-bc8b-e67f329c007f-kube-api-access-c7w2n" pod "45f0dda2-ba36-42a4-bc8b-e67f329c007f" (UID: "45f0dda2-ba36-42a4-bc8b-e67f329c007f"). InnerVolumeSpecName "kube-api-access-c7w2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:27.106166 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.106129    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45f0dda2-ba36-42a4-bc8b-e67f329c007f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:27.106166 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.106165    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7w2n\" (UniqueName: \"kubernetes.io/projected/45f0dda2-ba36-42a4-bc8b-e67f329c007f-kube-api-access-c7w2n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:27.722863 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.722835    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"
Oct 30 14:00:27.723138 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.722868    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg" event={"ID":"45f0dda2-ba36-42a4-bc8b-e67f329c007f","Type":"ContainerDied","Data":"33a90f14342c6455ee158801cd6e763ce59b68b6ea040e99fd15aa5a97bcbc58"}
Oct 30 14:00:27.723138 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.722944    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33a90f14342c6455ee158801cd6e763ce59b68b6ea040e99fd15aa5a97bcbc58"
Oct 30 14:00:27.880024 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:27.879992    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ade31a0f-9282-4bde-86a9-e17e7b41f05b" path="/var/lib/kubelet/pods/ade31a0f-9282-4bde-86a9-e17e7b41f05b/volumes"
Oct 30 14:00:33.595700 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:33.595645    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" podUID="6656cb26-447c-45e6-96c7-ab21cd9d6926" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:00:34.374727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.374691    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"]
Oct 30 14:00:34.375372 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375357    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:34.375434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375373    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:34.375434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375397    4291 cpu_manager.go:414] "containerMap: removing container" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:34.375434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375405    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421d7f0d-0d50-4985-95f4-4058c691b840" containerName="cleanup"
Oct 30 14:00:34.375434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375418    4291 cpu_manager.go:414] "containerMap: removing container" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:34.375434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375423    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851ad965-5bcf-46ed-bdf4-1764a61b901a" containerName="cleanup"
Oct 30 14:00:34.375434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375434    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375439    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375448    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375453    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375460    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375464    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375472    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375479    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375493    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375498    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375505    4291 cpu_manager.go:414] "containerMap: removing container" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375509    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375518    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375522    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375533    4291 cpu_manager.go:414] "containerMap: removing container" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375538    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375547    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375551    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375558    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375567    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375575    4291 cpu_manager.go:414] "containerMap: removing container" podUID="45f0dda2-ba36-42a4-bc8b-e67f329c007f" containerName="revoke-keys"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375580    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45f0dda2-ba36-42a4-bc8b-e67f329c007f" containerName="revoke-keys"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375588    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e68c3640-a5a8-4641-ab24-27c7f347b239" containerName="main"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375593    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e68c3640-a5a8-4641-ab24-27c7f347b239" containerName="main"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375600    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375605    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375611    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375616    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375626    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375646    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375656    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:34.375629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375660    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375669    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ade31a0f-9282-4bde-86a9-e17e7b41f05b" containerName="payload-processing"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375673    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ade31a0f-9282-4bde-86a9-e17e7b41f05b" containerName="payload-processing"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375683    4291 cpu_manager.go:414] "containerMap: removing container" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375688    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375695    4291 cpu_manager.go:414] "containerMap: removing container" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375699    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02c6941b-9f8e-4ca3-84cc-7fa377906a40" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375707    4291 cpu_manager.go:414] "containerMap: removing container" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375711    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375722    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375729    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375738    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375743    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375748    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375753    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f297ee2-5036-4ac9-8839-c80d2eb293e5" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375759    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375763    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375776    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375780    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375924    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375934    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375940    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="addb6a6e-6673-4881-b84c-fa9561235a47" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375948    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375958    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375970    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375976    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ade31a0f-9282-4bde-86a9-e17e7b41f05b" containerName="payload-processing"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375983    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375988    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375994    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="47367ecc-1223-4952-a0b8-986afe4951b9" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.375999    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376005    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="40ba6924-2d7d-41bc-9749-458e7a705e64" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376011    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376018    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376023    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376028    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e43b7f3d-c901-4f59-a039-04c48e60d984" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376037    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376045    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4fb266d-2bcf-4120-b82e-1f8ebefee7a6" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376051    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376061    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376066    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376072    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="2804875f-4c19-409b-8646-c17b8507f6f2" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376079    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf93e80a-eb0c-447c-a9c6-a37be5c77ab0" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376084    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0e7264f-66cf-45f6-91db-4126497c2ae6" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376089    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376095    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376100    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="45f0dda2-ba36-42a4-bc8b-e67f329c007f" containerName="revoke-keys"
Oct 30 14:00:34.376695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.376108    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e68c3640-a5a8-4641-ab24-27c7f347b239" containerName="main"
Oct 30 14:00:34.381119 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.381099    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.397802 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.397771    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"]
Oct 30 14:00:34.400566 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.400537    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:00:34.405079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.405042    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:00:34.484760 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.484718    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:00:34.484981 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.484896    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:00:34.545852 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.545812    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.546034 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.546007    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwcg\" (UniqueName: \"kubernetes.io/projected/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-kube-api-access-hdwcg\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.546170 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.546156    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.647089 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.647002    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwcg\" (UniqueName: \"kubernetes.io/projected/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-kube-api-access-hdwcg\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.647463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.647153    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.647463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.647221    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.647883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.647862    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.649626 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.649605    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.666319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.666293    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwcg\" (UniqueName: \"kubernetes.io/projected/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-kube-api-access-hdwcg\") pod \"maas-api-revoke-sub-f095babd-p7t7n\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:34.697615 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:34.697581    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:35.009405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:35.009372    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"]
Oct 30 14:00:35.010929 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:35.010899    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e24cc45_8b37_4863_8189_8d0dc5b4eecf.slice/crio-c4f3dc93f28eb9f9f17963d3daa931c40c0b5d145444e0459e5fb767454e3794 WatchSource:0}: Error finding container c4f3dc93f28eb9f9f17963d3daa931c40c0b5d145444e0459e5fb767454e3794: Status 404 returned error can't find the container with id c4f3dc93f28eb9f9f17963d3daa931c40c0b5d145444e0459e5fb767454e3794
Oct 30 14:00:35.836651 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:35.836605    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n" event={"ID":"5e24cc45-8b37-4863-8189-8d0dc5b4eecf","Type":"ContainerStarted","Data":"b01e29c6c37e0594fadb6a5fd0694135a4d19a32604c90fedae46d1fdfea5cfd"}
Oct 30 14:00:35.837062 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:35.836669    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n" event={"ID":"5e24cc45-8b37-4863-8189-8d0dc5b4eecf","Type":"ContainerStarted","Data":"c4f3dc93f28eb9f9f17963d3daa931c40c0b5d145444e0459e5fb767454e3794"}
Oct 30 14:00:35.875505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:35.875451    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n" podStartSLOduration=1.8754362709999999 podStartE2EDuration="1.875436271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:00:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:00:35.874370452 +0000 UTC m=+3446.134608686" watchObservedRunningTime="2026-10-30 14:00:35.875436271 +0000 UTC m=+3446.135674532"
Oct 30 14:00:36.846962 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:36.846925    4291 generic.go:356] "Generic (PLEG): container finished" podID="5e24cc45-8b37-4863-8189-8d0dc5b4eecf" containerID="b01e29c6c37e0594fadb6a5fd0694135a4d19a32604c90fedae46d1fdfea5cfd" exitCode=0
Oct 30 14:00:36.847413 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:36.847014    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n" event={"ID":"5e24cc45-8b37-4863-8189-8d0dc5b4eecf","Type":"ContainerDied","Data":"b01e29c6c37e0594fadb6a5fd0694135a4d19a32604c90fedae46d1fdfea5cfd"}
Oct 30 14:00:38.018007 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.017981    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:38.095287 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.095241    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-openshift-ca\") pod \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") "
Oct 30 14:00:38.095465 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.095432    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-kube-api-access-hdwcg\" (UniqueName: \"kubernetes.io/projected/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-kube-api-access-hdwcg\") pod \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") "
Oct 30 14:00:38.095518 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.095485    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-xks-ca\") pod \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\" (UID: \"5e24cc45-8b37-4863-8189-8d0dc5b4eecf\") "
Oct 30 14:00:38.095757 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.095723    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-openshift-ca" pod "5e24cc45-8b37-4863-8189-8d0dc5b4eecf" (UID: "5e24cc45-8b37-4863-8189-8d0dc5b4eecf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:38.096163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.096135    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:38.097602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.097576    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-xks-ca" pod "5e24cc45-8b37-4863-8189-8d0dc5b4eecf" (UID: "5e24cc45-8b37-4863-8189-8d0dc5b4eecf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:00:38.098144 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.098128    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-kube-api-access-hdwcg" pod "5e24cc45-8b37-4863-8189-8d0dc5b4eecf" (UID: "5e24cc45-8b37-4863-8189-8d0dc5b4eecf"). InnerVolumeSpecName "kube-api-access-hdwcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:38.197269 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.197231    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdwcg\" (UniqueName: \"kubernetes.io/projected/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-kube-api-access-hdwcg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:38.197269 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.197265    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e24cc45-8b37-4863-8189-8d0dc5b4eecf-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:38.214652 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:38.214611    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:00:38.866279 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.866248    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"
Oct 30 14:00:38.866495 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.866281    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n" event={"ID":"5e24cc45-8b37-4863-8189-8d0dc5b4eecf","Type":"ContainerDied","Data":"c4f3dc93f28eb9f9f17963d3daa931c40c0b5d145444e0459e5fb767454e3794"}
Oct 30 14:00:38.866495 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:38.866339    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4f3dc93f28eb9f9f17963d3daa931c40c0b5d145444e0459e5fb767454e3794"
Oct 30 14:00:43.595030 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:43.594989    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" podUID="6656cb26-447c-45e6-96c7-ab21cd9d6926" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:00:45.279084 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.279019    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 14:00:45.446599 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.446564    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6656cb26-447c-45e6-96c7-ab21cd9d6926-kube-api-access-j7l8m\" (UniqueName: \"kubernetes.io/projected/6656cb26-447c-45e6-96c7-ab21cd9d6926-kube-api-access-j7l8m\") pod \"6656cb26-447c-45e6-96c7-ab21cd9d6926\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") "
Oct 30 14:00:45.446787 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.446716    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6656cb26-447c-45e6-96c7-ab21cd9d6926-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6656cb26-447c-45e6-96c7-ab21cd9d6926-plugins-config-volume\") pod \"6656cb26-447c-45e6-96c7-ab21cd9d6926\" (UID: \"6656cb26-447c-45e6-96c7-ab21cd9d6926\") "
Oct 30 14:00:45.447106 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.447084    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6656cb26-447c-45e6-96c7-ab21cd9d6926-plugins-config-volume" pod "6656cb26-447c-45e6-96c7-ab21cd9d6926" (UID: "6656cb26-447c-45e6-96c7-ab21cd9d6926"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:45.447355 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.447340    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6656cb26-447c-45e6-96c7-ab21cd9d6926-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:45.448872 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.448854    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6656cb26-447c-45e6-96c7-ab21cd9d6926-kube-api-access-j7l8m" pod "6656cb26-447c-45e6-96c7-ab21cd9d6926" (UID: "6656cb26-447c-45e6-96c7-ab21cd9d6926"). InnerVolumeSpecName "kube-api-access-j7l8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:45.548765 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.548684    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7l8m\" (UniqueName: \"kubernetes.io/projected/6656cb26-447c-45e6-96c7-ab21cd9d6926-kube-api-access-j7l8m\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:45.937719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.937684    4291 generic.go:356] "Generic (PLEG): container finished" podID="6656cb26-447c-45e6-96c7-ab21cd9d6926" containerID="ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6" exitCode=0
Oct 30 14:00:45.937914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.937724    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"
Oct 30 14:00:45.937914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.937768    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" event={"ID":"6656cb26-447c-45e6-96c7-ab21cd9d6926","Type":"ContainerDied","Data":"ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6"}
Oct 30 14:00:45.937914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.937838    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh" event={"ID":"6656cb26-447c-45e6-96c7-ab21cd9d6926","Type":"ContainerDied","Data":"8a95d50e51c3440446141d56c0cb14872cc2ca51694caa67f8d9e8c42e3a21ff"}
Oct 30 14:00:45.937914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.937866    4291 scope.go:122] "RemoveContainer" containerID="ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6"
Oct 30 14:00:45.952854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.952836    4291 scope.go:122] "RemoveContainer" containerID="ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6"
Oct 30 14:00:45.953172 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:00:45.953146    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6\": container with ID starting with ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6 not found: ID does not exist" containerID="ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6"
Oct 30 14:00:45.953271 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.953179    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6"} err="failed to get container status \"ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6\": rpc error: code = NotFound desc = could not find container \"ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6\": container with ID starting with ab2ec57b5d05eebc3446453578b7e4bf8cac50ee6c8ccb8be6cfbc5922c152f6 not found: ID does not exist"
Oct 30 14:00:45.990257 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:45.990232    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"]
Oct 30 14:00:46.001831 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:46.001810    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-08c365-7568f6df4c-qgtgh"]
Oct 30 14:00:47.881250 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:47.881215    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6656cb26-447c-45e6-96c7-ab21cd9d6926" path="/var/lib/kubelet/pods/6656cb26-447c-45e6-96c7-ab21cd9d6926/volumes"
Oct 30 14:00:53.993472 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.993381    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"]
Oct 30 14:00:53.994149 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994131    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:53.994216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994152    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:53.994216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994178    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:53.994216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994185    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f764f2cc-15c4-460c-930c-9b16c9fa82ab" containerName="cleanup"
Oct 30 14:00:53.994216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994200    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:53.994216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994205    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994226    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994231    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994239    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5e24cc45-8b37-4863-8189-8d0dc5b4eecf" containerName="revoke-keys"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994243    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e24cc45-8b37-4863-8189-8d0dc5b4eecf" containerName="revoke-keys"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994258    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6656cb26-447c-45e6-96c7-ab21cd9d6926" containerName="payload-pre-processing"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994263    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6656cb26-447c-45e6-96c7-ab21cd9d6926" containerName="payload-pre-processing"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994270    4291 cpu_manager.go:414] "containerMap: removing container" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994275    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994284    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994288    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994302    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994307    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994402    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e24cc45-8b37-4863-8189-8d0dc5b4eecf" containerName="revoke-keys"
Oct 30 14:00:53.994405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994410    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6656cb26-447c-45e6-96c7-ab21cd9d6926" containerName="payload-pre-processing"
Oct 30 14:00:53.995070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994418    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="398137cb-99db-4931-bca0-f6e1b1b931a7" containerName="cleanup"
Oct 30 14:00:53.995070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994426    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d440f70-f544-432c-a8f1-e38d9cb7ec68" containerName="cleanup"
Oct 30 14:00:53.995070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994433    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ec2d9e9-61ec-411a-bcc8-1e7b7f6c849b" containerName="cleanup"
Oct 30 14:00:53.995070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994441    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5fb8d13-b0d8-439d-b7b2-9227b69604e7" containerName="cleanup"
Oct 30 14:00:53.995070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994447    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a630f92-dc84-454b-8f37-c80cdcfe40b1" containerName="cleanup"
Oct 30 14:00:53.995070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.994453    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc3d3284-1405-4e29-91fd-fba47df039df" containerName="cleanup"
Oct 30 14:00:53.997201 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:53.997186    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.014068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.014037    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"]
Oct 30 14:00:54.024482 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.024455    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:00:54.024663 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.024590    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:00:54.048748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.046274    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:00:54.048748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.046415    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:00:54.100947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.100904    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.101152 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.101044    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjt2z\" (UniqueName: \"kubernetes.io/projected/83f597e4-23ee-4b75-8e68-a9983e775cc3-kube-api-access-fjt2z\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.101152 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.101094    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.202457 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.202422    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.202701 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.202483    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjt2z\" (UniqueName: \"kubernetes.io/projected/83f597e4-23ee-4b75-8e68-a9983e775cc3-kube-api-access-fjt2z\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.202701 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.202514    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.203201 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.203175    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.204900 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.204875    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.222023 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.221993    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjt2z\" (UniqueName: \"kubernetes.io/projected/83f597e4-23ee-4b75-8e68-a9983e775cc3-kube-api-access-fjt2z\") pod \"maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.313274 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.313188    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:54.617940 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:54.617908    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"]
Oct 30 14:00:54.618928 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:00:54.618899    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f597e4_23ee_4b75_8e68_a9983e775cc3.slice/crio-2c3d91a2a009143d3721ae905f5c7e2383d0ca7249f3f4295736607b091e171a WatchSource:0}: Error finding container 2c3d91a2a009143d3721ae905f5c7e2383d0ca7249f3f4295736607b091e171a: Status 404 returned error can't find the container with id 2c3d91a2a009143d3721ae905f5c7e2383d0ca7249f3f4295736607b091e171a
Oct 30 14:00:55.206842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:55.206762    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx" event={"ID":"83f597e4-23ee-4b75-8e68-a9983e775cc3","Type":"ContainerStarted","Data":"6208fa556af864c6da9be1032df30d8d0aa5dfb051fb800faf79222e47ceca5c"}
Oct 30 14:00:55.206842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:55.206814    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx" event={"ID":"83f597e4-23ee-4b75-8e68-a9983e775cc3","Type":"ContainerStarted","Data":"2c3d91a2a009143d3721ae905f5c7e2383d0ca7249f3f4295736607b091e171a"}
Oct 30 14:00:56.217233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:56.217197    4291 generic.go:356] "Generic (PLEG): container finished" podID="83f597e4-23ee-4b75-8e68-a9983e775cc3" containerID="6208fa556af864c6da9be1032df30d8d0aa5dfb051fb800faf79222e47ceca5c" exitCode=0
Oct 30 14:00:56.217677 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:56.217294    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx" event={"ID":"83f597e4-23ee-4b75-8e68-a9983e775cc3","Type":"ContainerDied","Data":"6208fa556af864c6da9be1032df30d8d0aa5dfb051fb800faf79222e47ceca5c"}
Oct 30 14:00:57.394591 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.394566    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:57.444929 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.444887    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83f597e4-23ee-4b75-8e68-a9983e775cc3-kube-api-access-fjt2z\" (UniqueName: \"kubernetes.io/projected/83f597e4-23ee-4b75-8e68-a9983e775cc3-kube-api-access-fjt2z\") pod \"83f597e4-23ee-4b75-8e68-a9983e775cc3\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") "
Oct 30 14:00:57.445111 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.445050    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-openshift-ca\") pod \"83f597e4-23ee-4b75-8e68-a9983e775cc3\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") "
Oct 30 14:00:57.445237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.445215    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-xks-ca\") pod \"83f597e4-23ee-4b75-8e68-a9983e775cc3\" (UID: \"83f597e4-23ee-4b75-8e68-a9983e775cc3\") "
Oct 30 14:00:57.445982 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.445954    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-openshift-ca" pod "83f597e4-23ee-4b75-8e68-a9983e775cc3" (UID: "83f597e4-23ee-4b75-8e68-a9983e775cc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:00:57.450717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.450690    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83f597e4-23ee-4b75-8e68-a9983e775cc3-kube-api-access-fjt2z" pod "83f597e4-23ee-4b75-8e68-a9983e775cc3" (UID: "83f597e4-23ee-4b75-8e68-a9983e775cc3"). InnerVolumeSpecName "kube-api-access-fjt2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:00:57.451742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.451714    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-xks-ca" pod "83f597e4-23ee-4b75-8e68-a9983e775cc3" (UID: "83f597e4-23ee-4b75-8e68-a9983e775cc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:00:57.546978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.546894    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:57.546978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.546926    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjt2z\" (UniqueName: \"kubernetes.io/projected/83f597e4-23ee-4b75-8e68-a9983e775cc3-kube-api-access-fjt2z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:57.546978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:57.546941    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83f597e4-23ee-4b75-8e68-a9983e775cc3-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:00:58.235159 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:58.235090    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"
Oct 30 14:00:58.235310 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:58.235085    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx" event={"ID":"83f597e4-23ee-4b75-8e68-a9983e775cc3","Type":"ContainerDied","Data":"2c3d91a2a009143d3721ae905f5c7e2383d0ca7249f3f4295736607b091e171a"}
Oct 30 14:00:58.235310 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:00:58.235213    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c3d91a2a009143d3721ae905f5c7e2383d0ca7249f3f4295736607b091e171a"
Oct 30 14:01:01.631587 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.631550    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"]
Oct 30 14:01:01.632490 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.632467    4291 cpu_manager.go:414] "containerMap: removing container" podUID="83f597e4-23ee-4b75-8e68-a9983e775cc3" containerName="revoke-keys"
Oct 30 14:01:01.632593 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.632492    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83f597e4-23ee-4b75-8e68-a9983e775cc3" containerName="revoke-keys"
Oct 30 14:01:01.632716 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.632701    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="83f597e4-23ee-4b75-8e68-a9983e775cc3" containerName="revoke-keys"
Oct 30 14:01:01.637147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.637125    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.652911 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.652882    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"]
Oct 30 14:01:01.658354 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.658264    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:01:01.658354 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.658321    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:01:01.676403 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.676228    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:01:01.677090 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.677064    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:01:01.718828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.718770    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.719056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.718908    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.719056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.718935    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9tlv\" (UniqueName: \"kubernetes.io/projected/77f274c8-4a15-4659-9720-e2a603b5e31e-kube-api-access-c9tlv\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.820402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.820365    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.820589 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.820453    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.820589 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.820486    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9tlv\" (UniqueName: \"kubernetes.io/projected/77f274c8-4a15-4659-9720-e2a603b5e31e-kube-api-access-c9tlv\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.821417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.821390    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.823163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.823137    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.840600 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.840568    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9tlv\" (UniqueName: \"kubernetes.io/projected/77f274c8-4a15-4659-9720-e2a603b5e31e-kube-api-access-c9tlv\") pod \"maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:01.956408 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:01.956311    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:02.267749 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:02.267720    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"]
Oct 30 14:01:02.268371 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:01:02.268343    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77f274c8_4a15_4659_9720_e2a603b5e31e.slice/crio-085cf0f18070d6b7bb8d978d7f4e48d580f57ea26291bd34a0364b43bf810496 WatchSource:0}: Error finding container 085cf0f18070d6b7bb8d978d7f4e48d580f57ea26291bd34a0364b43bf810496: Status 404 returned error can't find the container with id 085cf0f18070d6b7bb8d978d7f4e48d580f57ea26291bd34a0364b43bf810496
Oct 30 14:01:02.290005 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:02.289970    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr" event={"ID":"77f274c8-4a15-4659-9720-e2a603b5e31e","Type":"ContainerStarted","Data":"085cf0f18070d6b7bb8d978d7f4e48d580f57ea26291bd34a0364b43bf810496"}
Oct 30 14:01:03.299944 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:03.299842    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr" event={"ID":"77f274c8-4a15-4659-9720-e2a603b5e31e","Type":"ContainerStarted","Data":"0c8570be346c73c27282e8dc9c730f61690064627241112df82d67d99e0f0a23"}
Oct 30 14:01:03.346847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:03.346783    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr" podStartSLOduration=2.346764565 podStartE2EDuration="2.346764565s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:01:03.33995465 +0000 UTC m=+3473.600192886" watchObservedRunningTime="2026-10-30 14:01:03.346764565 +0000 UTC m=+3473.607002801"
Oct 30 14:01:04.308716 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:04.308677    4291 generic.go:356] "Generic (PLEG): container finished" podID="77f274c8-4a15-4659-9720-e2a603b5e31e" containerID="0c8570be346c73c27282e8dc9c730f61690064627241112df82d67d99e0f0a23" exitCode=0
Oct 30 14:01:04.309151 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:04.308763    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr" event={"ID":"77f274c8-4a15-4659-9720-e2a603b5e31e","Type":"ContainerDied","Data":"0c8570be346c73c27282e8dc9c730f61690064627241112df82d67d99e0f0a23"}
Oct 30 14:01:04.666213 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:04.666180    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"]
Oct 30 14:01:04.697997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:04.697958    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"]
Oct 30 14:01:05.598224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.598158    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:05.784033 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.783993    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-openshift-ca\") pod \"77f274c8-4a15-4659-9720-e2a603b5e31e\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") "
Oct 30 14:01:05.784242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.784057    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77f274c8-4a15-4659-9720-e2a603b5e31e-kube-api-access-c9tlv\" (UniqueName: \"kubernetes.io/projected/77f274c8-4a15-4659-9720-e2a603b5e31e-kube-api-access-c9tlv\") pod \"77f274c8-4a15-4659-9720-e2a603b5e31e\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") "
Oct 30 14:01:05.784242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.784179    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-xks-ca\") pod \"77f274c8-4a15-4659-9720-e2a603b5e31e\" (UID: \"77f274c8-4a15-4659-9720-e2a603b5e31e\") "
Oct 30 14:01:05.784451 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.784429    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-openshift-ca" pod "77f274c8-4a15-4659-9720-e2a603b5e31e" (UID: "77f274c8-4a15-4659-9720-e2a603b5e31e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:05.784748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.784726    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:05.786346 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.786316    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-xks-ca" pod "77f274c8-4a15-4659-9720-e2a603b5e31e" (UID: "77f274c8-4a15-4659-9720-e2a603b5e31e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:01:05.786462 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.786385    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77f274c8-4a15-4659-9720-e2a603b5e31e-kube-api-access-c9tlv" pod "77f274c8-4a15-4659-9720-e2a603b5e31e" (UID: "77f274c8-4a15-4659-9720-e2a603b5e31e"). InnerVolumeSpecName "kube-api-access-c9tlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:05.885860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.885742    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9tlv\" (UniqueName: \"kubernetes.io/projected/77f274c8-4a15-4659-9720-e2a603b5e31e-kube-api-access-c9tlv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:05.885860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:05.885781    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77f274c8-4a15-4659-9720-e2a603b5e31e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:06.353359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:06.353331    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"
Oct 30 14:01:06.353535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:06.353331    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr" event={"ID":"77f274c8-4a15-4659-9720-e2a603b5e31e","Type":"ContainerDied","Data":"085cf0f18070d6b7bb8d978d7f4e48d580f57ea26291bd34a0364b43bf810496"}
Oct 30 14:01:06.353535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:06.353458    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="085cf0f18070d6b7bb8d978d7f4e48d580f57ea26291bd34a0364b43bf810496"
Oct 30 14:01:08.212799 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:01:08.212738    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:01:09.666725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:09.666684    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g" podUID="4b822f93-3cd3-411d-a127-fe2df056c84b" containerName="payload-processing" containerID="cri-o://d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0" gracePeriod=25
Oct 30 14:01:09.698563 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:09.698505    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" podUID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" containerName="payload-pre-processing" containerID="cri-o://abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8" gracePeriod=25
Oct 30 14:01:09.957007 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:09.956980    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 14:01:10.041828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.041771    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b822f93-3cd3-411d-a127-fe2df056c84b-kube-api-access-zglbk\" (UniqueName: \"kubernetes.io/projected/4b822f93-3cd3-411d-a127-fe2df056c84b-kube-api-access-zglbk\") pod \"4b822f93-3cd3-411d-a127-fe2df056c84b\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") "
Oct 30 14:01:10.042056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.041926    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b822f93-3cd3-411d-a127-fe2df056c84b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b822f93-3cd3-411d-a127-fe2df056c84b-plugins-config-volume\") pod \"4b822f93-3cd3-411d-a127-fe2df056c84b\" (UID: \"4b822f93-3cd3-411d-a127-fe2df056c84b\") "
Oct 30 14:01:10.042872 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.042840    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b822f93-3cd3-411d-a127-fe2df056c84b-plugins-config-volume" pod "4b822f93-3cd3-411d-a127-fe2df056c84b" (UID: "4b822f93-3cd3-411d-a127-fe2df056c84b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:10.045226 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.045196    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b822f93-3cd3-411d-a127-fe2df056c84b-kube-api-access-zglbk" pod "4b822f93-3cd3-411d-a127-fe2df056c84b" (UID: "4b822f93-3cd3-411d-a127-fe2df056c84b"). InnerVolumeSpecName "kube-api-access-zglbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:10.143995 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.143960    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zglbk\" (UniqueName: \"kubernetes.io/projected/4b822f93-3cd3-411d-a127-fe2df056c84b-kube-api-access-zglbk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:10.143995 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.143990    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b822f93-3cd3-411d-a127-fe2df056c84b-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:10.386868 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.386836    4291 generic.go:356] "Generic (PLEG): container finished" podID="4b822f93-3cd3-411d-a127-fe2df056c84b" containerID="d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0" exitCode=0
Oct 30 14:01:10.387070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.386883    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"
Oct 30 14:01:10.387070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.386922    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g" event={"ID":"4b822f93-3cd3-411d-a127-fe2df056c84b","Type":"ContainerDied","Data":"d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0"}
Oct 30 14:01:10.387070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.386979    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g" event={"ID":"4b822f93-3cd3-411d-a127-fe2df056c84b","Type":"ContainerDied","Data":"68ac440e5d0b4f0ba2b0fd3c4659067a90551bbf9444b3341d4e356b0a98b0af"}
Oct 30 14:01:10.387070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.387003    4291 scope.go:122] "RemoveContainer" containerID="d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0"
Oct 30 14:01:10.398484 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.398461    4291 scope.go:122] "RemoveContainer" containerID="d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0"
Oct 30 14:01:10.398782 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:01:10.398762    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0\": container with ID starting with d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0 not found: ID does not exist" containerID="d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0"
Oct 30 14:01:10.398883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.398794    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0"} err="failed to get container status \"d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0\": rpc error: code = NotFound desc = could not find container \"d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0\": container with ID starting with d2f692735053b993bea68bb21d33bdaa1f730fc379fb777ce21be43ab52832e0 not found: ID does not exist"
Oct 30 14:01:10.439053 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.439017    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"]
Oct 30 14:01:10.443021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:10.442991    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-926ff0-677747c746-9bs4g"]
Oct 30 14:01:11.880153 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:11.880113    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b822f93-3cd3-411d-a127-fe2df056c84b" path="/var/lib/kubelet/pods/4b822f93-3cd3-411d-a127-fe2df056c84b/volumes"
Oct 30 14:01:12.073524 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:12.073489    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"]
Oct 30 14:01:12.106753 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:12.106719    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"]
Oct 30 14:01:17.073982 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.073911    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs" podUID="8d3b37cb-3117-4dce-9f23-b9eae6eea81f" containerName="payload-processing" containerID="cri-o://b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156" gracePeriod=25
Oct 30 14:01:17.108099 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.108051    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" podUID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" containerName="payload-pre-processing" containerID="cri-o://d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed" gracePeriod=25
Oct 30 14:01:17.139751 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.139712    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" podUID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:01:17.423138 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.423109    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 14:01:17.450731 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.450691    4291 generic.go:356] "Generic (PLEG): container finished" podID="8d3b37cb-3117-4dce-9f23-b9eae6eea81f" containerID="b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156" exitCode=0
Oct 30 14:01:17.450961 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.450744    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"
Oct 30 14:01:17.450961 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.450780    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs" event={"ID":"8d3b37cb-3117-4dce-9f23-b9eae6eea81f","Type":"ContainerDied","Data":"b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156"}
Oct 30 14:01:17.450961 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.450843    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs" event={"ID":"8d3b37cb-3117-4dce-9f23-b9eae6eea81f","Type":"ContainerDied","Data":"56ef4c4680a3d30a1716cbe4d7b1f99f021f5c4fcee373bd665d3d8682cdda69"}
Oct 30 14:01:17.450961 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.450877    4291 scope.go:122] "RemoveContainer" containerID="b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156"
Oct 30 14:01:17.465179 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.465144    4291 scope.go:122] "RemoveContainer" containerID="b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156"
Oct 30 14:01:17.469051 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:01:17.469017    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156\": container with ID starting with b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156 not found: ID does not exist" containerID="b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156"
Oct 30 14:01:17.469218 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.469062    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156"} err="failed to get container status \"b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156\": rpc error: code = NotFound desc = could not find container \"b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156\": container with ID starting with b02d62a6e7d58239a7158f85cf9ff3cc2959eb2014957820a8cb7d9cb477d156 not found: ID does not exist"
Oct 30 14:01:17.560903 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.560867    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-kube-api-access-6jcqh\" (UniqueName: \"kubernetes.io/projected/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-kube-api-access-6jcqh\") pod \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") "
Oct 30 14:01:17.561099 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.560927    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-plugins-config-volume\") pod \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\" (UID: \"8d3b37cb-3117-4dce-9f23-b9eae6eea81f\") "
Oct 30 14:01:17.561354 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.561326    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-plugins-config-volume" pod "8d3b37cb-3117-4dce-9f23-b9eae6eea81f" (UID: "8d3b37cb-3117-4dce-9f23-b9eae6eea81f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:17.561534 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.561514    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:17.563146 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.563118    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-kube-api-access-6jcqh" pod "8d3b37cb-3117-4dce-9f23-b9eae6eea81f" (UID: "8d3b37cb-3117-4dce-9f23-b9eae6eea81f"). InnerVolumeSpecName "kube-api-access-6jcqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:17.663767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.663671    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jcqh\" (UniqueName: \"kubernetes.io/projected/8d3b37cb-3117-4dce-9f23-b9eae6eea81f-kube-api-access-6jcqh\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:17.799559 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.799506    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"]
Oct 30 14:01:17.809995 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.809958    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-09014b-5bbc4db8c4-nk4cs"]
Oct 30 14:01:17.880095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:17.880062    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8d3b37cb-3117-4dce-9f23-b9eae6eea81f" path="/var/lib/kubelet/pods/8d3b37cb-3117-4dce-9f23-b9eae6eea81f/volumes"
Oct 30 14:01:22.827836 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:22.827788    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" podUID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:01:27.139164 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:27.139116    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" podUID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:01:28.840252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:28.840218    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:01:29.978779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:29.978755    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 14:01:30.055873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.055838    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-kube-api-access-rkkjs\" (UniqueName: \"kubernetes.io/projected/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-kube-api-access-rkkjs\") pod \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") "
Oct 30 14:01:30.056067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.056027    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-plugins-config-volume\") pod \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\" (UID: \"9e806a3c-b759-4e87-89b7-9bb0a7c911c9\") "
Oct 30 14:01:30.056418 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.056386    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-plugins-config-volume" pod "9e806a3c-b759-4e87-89b7-9bb0a7c911c9" (UID: "9e806a3c-b759-4e87-89b7-9bb0a7c911c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:30.056702 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.056677    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:30.058418 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.058388    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-kube-api-access-rkkjs" pod "9e806a3c-b759-4e87-89b7-9bb0a7c911c9" (UID: "9e806a3c-b759-4e87-89b7-9bb0a7c911c9"). InnerVolumeSpecName "kube-api-access-rkkjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:30.157597 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.157563    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkkjs\" (UniqueName: \"kubernetes.io/projected/9e806a3c-b759-4e87-89b7-9bb0a7c911c9-kube-api-access-rkkjs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:30.551329 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.551291    4291 generic.go:356] "Generic (PLEG): container finished" podID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" containerID="abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8" exitCode=0
Oct 30 14:01:30.551515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.551339    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"
Oct 30 14:01:30.551515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.551363    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" event={"ID":"9e806a3c-b759-4e87-89b7-9bb0a7c911c9","Type":"ContainerDied","Data":"abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8"}
Oct 30 14:01:30.551515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.551415    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh" event={"ID":"9e806a3c-b759-4e87-89b7-9bb0a7c911c9","Type":"ContainerDied","Data":"c1e56bec41d4fa51ebdc43a3b37e04ea1c9efa56e2755129f660d9306febda17"}
Oct 30 14:01:30.551515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.551437    4291 scope.go:122] "RemoveContainer" containerID="abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8"
Oct 30 14:01:30.563690 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.563671    4291 scope.go:122] "RemoveContainer" containerID="abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8"
Oct 30 14:01:30.563970 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:01:30.563947    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8\": container with ID starting with abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8 not found: ID does not exist" containerID="abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8"
Oct 30 14:01:30.564037 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.563981    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8"} err="failed to get container status \"abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8\": rpc error: code = NotFound desc = could not find container \"abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8\": container with ID starting with abbdb5b94090a2e242fa3853e411100262466ac0100b64db58a94924610113a8 not found: ID does not exist"
Oct 30 14:01:30.620271 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.620234    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"]
Oct 30 14:01:30.623923 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:30.623896    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-926ff0-6475f89747-ww8hh"]
Oct 30 14:01:31.884045 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:31.884007    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" path="/var/lib/kubelet/pods/9e806a3c-b759-4e87-89b7-9bb0a7c911c9/volumes"
Oct 30 14:01:32.827415 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:32.827380    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" podUID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:01:37.427416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.427009    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 14:01:37.570353 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.570322    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-kube-api-access-p8rd8\" (UniqueName: \"kubernetes.io/projected/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-kube-api-access-p8rd8\") pod \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") "
Oct 30 14:01:37.570543 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.570393    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-plugins-config-volume\") pod \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\" (UID: \"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a\") "
Oct 30 14:01:37.570763 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.570736    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-plugins-config-volume" pod "4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" (UID: "4d4333dd-8bbb-4304-abe9-f6a1fadbe36a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:37.570920 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.570906    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:37.572434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.572407    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-kube-api-access-p8rd8" pod "4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" (UID: "4d4333dd-8bbb-4304-abe9-f6a1fadbe36a"). InnerVolumeSpecName "kube-api-access-p8rd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:37.616267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.616236    4291 generic.go:356] "Generic (PLEG): container finished" podID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" containerID="d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed" exitCode=0
Oct 30 14:01:37.616416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.616278    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"
Oct 30 14:01:37.616416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.616319    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" event={"ID":"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a","Type":"ContainerDied","Data":"d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed"}
Oct 30 14:01:37.616416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.616376    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl" event={"ID":"4d4333dd-8bbb-4304-abe9-f6a1fadbe36a","Type":"ContainerDied","Data":"1761ce4eacdba8783247209a0f128239cc1529a02c8e008d999cf7e46dfe0701"}
Oct 30 14:01:37.616416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.616410    4291 scope.go:122] "RemoveContainer" containerID="d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed"
Oct 30 14:01:37.627214 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.627199    4291 scope.go:122] "RemoveContainer" containerID="d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed"
Oct 30 14:01:37.627458 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:01:37.627440    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed\": container with ID starting with d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed not found: ID does not exist" containerID="d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed"
Oct 30 14:01:37.627509 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.627465    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed"} err="failed to get container status \"d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed\": rpc error: code = NotFound desc = could not find container \"d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed\": container with ID starting with d37a8ac6e7b8d084d68493d95baffe6668e66bef2fdeb53658893e02a77093ed not found: ID does not exist"
Oct 30 14:01:37.670069 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.670042    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"]
Oct 30 14:01:37.672448 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.672427    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8rd8\" (UniqueName: \"kubernetes.io/projected/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a-kube-api-access-p8rd8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:37.673782 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.673764    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-09014b-75f7685f5f-xm9jl"]
Oct 30 14:01:37.881202 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:37.881123    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" path="/var/lib/kubelet/pods/4d4333dd-8bbb-4304-abe9-f6a1fadbe36a/volumes"
Oct 30 14:01:38.212069 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:01:38.211984    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:01:39.324880 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.324831    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"]
Oct 30 14:01:39.325530 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325515    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4b822f93-3cd3-411d-a127-fe2df056c84b" containerName="payload-processing"
Oct 30 14:01:39.325576 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325531    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b822f93-3cd3-411d-a127-fe2df056c84b" containerName="payload-processing"
Oct 30 14:01:39.325576 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325553    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8d3b37cb-3117-4dce-9f23-b9eae6eea81f" containerName="payload-processing"
Oct 30 14:01:39.325576 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325558    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8d3b37cb-3117-4dce-9f23-b9eae6eea81f" containerName="payload-processing"
Oct 30 14:01:39.325734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325579    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" containerName="payload-pre-processing"
Oct 30 14:01:39.325734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325586    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" containerName="payload-pre-processing"
Oct 30 14:01:39.325734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325595    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" containerName="payload-pre-processing"
Oct 30 14:01:39.325734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325600    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" containerName="payload-pre-processing"
Oct 30 14:01:39.325734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325609    4291 cpu_manager.go:414] "containerMap: removing container" podUID="77f274c8-4a15-4659-9720-e2a603b5e31e" containerName="revoke-keys"
Oct 30 14:01:39.325734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325614    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77f274c8-4a15-4659-9720-e2a603b5e31e" containerName="revoke-keys"
Oct 30 14:01:39.325931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325736    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b822f93-3cd3-411d-a127-fe2df056c84b" containerName="payload-processing"
Oct 30 14:01:39.325931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325749    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8d3b37cb-3117-4dce-9f23-b9eae6eea81f" containerName="payload-processing"
Oct 30 14:01:39.325931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325759    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e806a3c-b759-4e87-89b7-9bb0a7c911c9" containerName="payload-pre-processing"
Oct 30 14:01:39.325931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325767    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d4333dd-8bbb-4304-abe9-f6a1fadbe36a" containerName="payload-pre-processing"
Oct 30 14:01:39.325931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.325775    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="77f274c8-4a15-4659-9720-e2a603b5e31e" containerName="revoke-keys"
Oct 30 14:01:39.330916 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.330897    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.346727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.346698    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"]
Oct 30 14:01:39.355219 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.353393    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:01:39.355219 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.353455    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:01:39.355219 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.353401    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:01:39.356734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.356707    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:01:39.396680 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.396598    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.396893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.396762    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9glx\" (UniqueName: \"kubernetes.io/projected/f672a605-4281-49a2-8bd9-b3806eb18578-kube-api-access-b9glx\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.397039 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.397017    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.498644 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.498600    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.498877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.498730    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.498877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.498807    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9glx\" (UniqueName: \"kubernetes.io/projected/f672a605-4281-49a2-8bd9-b3806eb18578-kube-api-access-b9glx\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.499310 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.499291    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.501126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.501103    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.518381 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.518349    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9glx\" (UniqueName: \"kubernetes.io/projected/f672a605-4281-49a2-8bd9-b3806eb18578-kube-api-access-b9glx\") pod \"maas-api-revoke-sub-73188bb4-jxch5\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.647594 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.647558    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:39.953425 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:39.953392    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"]
Oct 30 14:01:39.954161 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:01:39.954134    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf672a605_4281_49a2_8bd9_b3806eb18578.slice/crio-cb6d3eded1dac8294f56ebb708bcc7b6ccda98c4ba2b7114e446ae4cfca6b34b WatchSource:0}: Error finding container cb6d3eded1dac8294f56ebb708bcc7b6ccda98c4ba2b7114e446ae4cfca6b34b: Status 404 returned error can't find the container with id cb6d3eded1dac8294f56ebb708bcc7b6ccda98c4ba2b7114e446ae4cfca6b34b
Oct 30 14:01:40.641305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:40.641264    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5" event={"ID":"f672a605-4281-49a2-8bd9-b3806eb18578","Type":"ContainerStarted","Data":"540e88d468f2aab99d217ffe8a72a02d8affb454bfc5b8fa5b7006b20393fc6e"}
Oct 30 14:01:40.642137 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:40.641355    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5" event={"ID":"f672a605-4281-49a2-8bd9-b3806eb18578","Type":"ContainerStarted","Data":"cb6d3eded1dac8294f56ebb708bcc7b6ccda98c4ba2b7114e446ae4cfca6b34b"}
Oct 30 14:01:41.650270 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:41.650232    4291 generic.go:356] "Generic (PLEG): container finished" podID="f672a605-4281-49a2-8bd9-b3806eb18578" containerID="540e88d468f2aab99d217ffe8a72a02d8affb454bfc5b8fa5b7006b20393fc6e" exitCode=0
Oct 30 14:01:41.650727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:41.650318    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5" event={"ID":"f672a605-4281-49a2-8bd9-b3806eb18578","Type":"ContainerDied","Data":"540e88d468f2aab99d217ffe8a72a02d8affb454bfc5b8fa5b7006b20393fc6e"}
Oct 30 14:01:42.828800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.828770    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:42.945104 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.944995    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-openshift-ca\") pod \"f672a605-4281-49a2-8bd9-b3806eb18578\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") "
Oct 30 14:01:42.945311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.945146    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f672a605-4281-49a2-8bd9-b3806eb18578-kube-api-access-b9glx\" (UniqueName: \"kubernetes.io/projected/f672a605-4281-49a2-8bd9-b3806eb18578-kube-api-access-b9glx\") pod \"f672a605-4281-49a2-8bd9-b3806eb18578\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") "
Oct 30 14:01:42.945311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.945252    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-xks-ca\") pod \"f672a605-4281-49a2-8bd9-b3806eb18578\" (UID: \"f672a605-4281-49a2-8bd9-b3806eb18578\") "
Oct 30 14:01:42.945499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.945466    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-openshift-ca" pod "f672a605-4281-49a2-8bd9-b3806eb18578" (UID: "f672a605-4281-49a2-8bd9-b3806eb18578"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:42.945982 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.945933    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:42.947383 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.947362    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-xks-ca" pod "f672a605-4281-49a2-8bd9-b3806eb18578" (UID: "f672a605-4281-49a2-8bd9-b3806eb18578"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:01:42.947466 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:42.947439    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f672a605-4281-49a2-8bd9-b3806eb18578-kube-api-access-b9glx" pod "f672a605-4281-49a2-8bd9-b3806eb18578" (UID: "f672a605-4281-49a2-8bd9-b3806eb18578"). InnerVolumeSpecName "kube-api-access-b9glx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:43.047656 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:43.047607    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9glx\" (UniqueName: \"kubernetes.io/projected/f672a605-4281-49a2-8bd9-b3806eb18578-kube-api-access-b9glx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:43.047834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:43.047684    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f672a605-4281-49a2-8bd9-b3806eb18578-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:43.667666 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:43.667604    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"
Oct 30 14:01:43.667875 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:43.667657    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5" event={"ID":"f672a605-4281-49a2-8bd9-b3806eb18578","Type":"ContainerDied","Data":"cb6d3eded1dac8294f56ebb708bcc7b6ccda98c4ba2b7114e446ae4cfca6b34b"}
Oct 30 14:01:43.667875 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:43.667722    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb6d3eded1dac8294f56ebb708bcc7b6ccda98c4ba2b7114e446ae4cfca6b34b"
Oct 30 14:01:45.930493 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:45.930460    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"]
Oct 30 14:01:45.934161 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:45.934132    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-926ff0-lkzgx"]
Oct 30 14:01:47.883510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:47.883467    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83f597e4-23ee-4b75-8e68-a9983e775cc3" path="/var/lib/kubelet/pods/83f597e4-23ee-4b75-8e68-a9983e775cc3/volumes"
Oct 30 14:01:53.179440 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:53.179395    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"]
Oct 30 14:01:53.191148 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:53.191110    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-09014b-dsvfr"]
Oct 30 14:01:53.881079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:53.881042    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77f274c8-4a15-4659-9720-e2a603b5e31e" path="/var/lib/kubelet/pods/77f274c8-4a15-4659-9720-e2a603b5e31e/volumes"
Oct 30 14:01:54.302553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.302471    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"]
Oct 30 14:01:54.303150 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.303134    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f672a605-4281-49a2-8bd9-b3806eb18578" containerName="revoke-keys"
Oct 30 14:01:54.303196 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.303152    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f672a605-4281-49a2-8bd9-b3806eb18578" containerName="revoke-keys"
Oct 30 14:01:54.303288 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.303278    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f672a605-4281-49a2-8bd9-b3806eb18578" containerName="revoke-keys"
Oct 30 14:01:54.307083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.307066    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.322855 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.322825    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"]
Oct 30 14:01:54.330136 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.330108    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:01:54.330310 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.330191    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:01:54.330310 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.330198    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:01:54.333341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.333317    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:01:54.376338 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.376303    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.376514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.376436    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.376514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.376501    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdv5f\" (UniqueName: \"kubernetes.io/projected/ccc977e5-25f8-4ba3-8280-b751da1d05f8-kube-api-access-fdv5f\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.477965 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.477917    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdv5f\" (UniqueName: \"kubernetes.io/projected/ccc977e5-25f8-4ba3-8280-b751da1d05f8-kube-api-access-fdv5f\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.478198 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.478072    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.478198 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.478180    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.478942 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.478912    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.480917 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.480889    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.499858 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.499818    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdv5f\" (UniqueName: \"kubernetes.io/projected/ccc977e5-25f8-4ba3-8280-b751da1d05f8-kube-api-access-fdv5f\") pod \"maas-api-revoke-keys-e2e-inf-b-319179-vb464\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.622733 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.622692    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:54.932582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:54.932553    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"]
Oct 30 14:01:54.933185 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:01:54.933160    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc977e5_25f8_4ba3_8280_b751da1d05f8.slice/crio-0189dd5a37475c527971f77cb6be9b96972270e421ad78454761bfd75d715694 WatchSource:0}: Error finding container 0189dd5a37475c527971f77cb6be9b96972270e421ad78454761bfd75d715694: Status 404 returned error can't find the container with id 0189dd5a37475c527971f77cb6be9b96972270e421ad78454761bfd75d715694
Oct 30 14:01:55.836509 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:55.836417    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464" event={"ID":"ccc977e5-25f8-4ba3-8280-b751da1d05f8","Type":"ContainerStarted","Data":"12d30dc58adfd15f446cc99cf2528dd4af3d565aac3417c9b83516144ce1f612"}
Oct 30 14:01:55.836509 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:55.836473    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464" event={"ID":"ccc977e5-25f8-4ba3-8280-b751da1d05f8","Type":"ContainerStarted","Data":"0189dd5a37475c527971f77cb6be9b96972270e421ad78454761bfd75d715694"}
Oct 30 14:01:55.892410 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:55.892353    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464" podStartSLOduration=1.892335406 podStartE2EDuration="1.892335406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:01:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:01:55.887845531 +0000 UTC m=+3526.148083766" watchObservedRunningTime="2026-10-30 14:01:55.892335406 +0000 UTC m=+3526.152573640"
Oct 30 14:01:56.845741 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:56.845702    4291 generic.go:356] "Generic (PLEG): container finished" podID="ccc977e5-25f8-4ba3-8280-b751da1d05f8" containerID="12d30dc58adfd15f446cc99cf2528dd4af3d565aac3417c9b83516144ce1f612" exitCode=0
Oct 30 14:01:56.846154 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:56.845791    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464" event={"ID":"ccc977e5-25f8-4ba3-8280-b751da1d05f8","Type":"ContainerDied","Data":"12d30dc58adfd15f446cc99cf2528dd4af3d565aac3417c9b83516144ce1f612"}
Oct 30 14:01:58.018595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.018564    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:58.147853 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.147736    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc977e5-25f8-4ba3-8280-b751da1d05f8-kube-api-access-fdv5f\" (UniqueName: \"kubernetes.io/projected/ccc977e5-25f8-4ba3-8280-b751da1d05f8-kube-api-access-fdv5f\") pod \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") "
Oct 30 14:01:58.148031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.147888    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-openshift-ca\") pod \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") "
Oct 30 14:01:58.148071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.148043    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-xks-ca\") pod \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\" (UID: \"ccc977e5-25f8-4ba3-8280-b751da1d05f8\") "
Oct 30 14:01:58.148279 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.148245    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-openshift-ca" pod "ccc977e5-25f8-4ba3-8280-b751da1d05f8" (UID: "ccc977e5-25f8-4ba3-8280-b751da1d05f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:01:58.148514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.148500    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:58.150102 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.150075    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-xks-ca" pod "ccc977e5-25f8-4ba3-8280-b751da1d05f8" (UID: "ccc977e5-25f8-4ba3-8280-b751da1d05f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:01:58.150209 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.150139    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc977e5-25f8-4ba3-8280-b751da1d05f8-kube-api-access-fdv5f" pod "ccc977e5-25f8-4ba3-8280-b751da1d05f8" (UID: "ccc977e5-25f8-4ba3-8280-b751da1d05f8"). InnerVolumeSpecName "kube-api-access-fdv5f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:01:58.249445 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.249405    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ccc977e5-25f8-4ba3-8280-b751da1d05f8-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:58.249445 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.249439    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdv5f\" (UniqueName: \"kubernetes.io/projected/ccc977e5-25f8-4ba3-8280-b751da1d05f8-kube-api-access-fdv5f\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:01:58.862804 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.862766    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"
Oct 30 14:01:58.863012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.862802    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464" event={"ID":"ccc977e5-25f8-4ba3-8280-b751da1d05f8","Type":"ContainerDied","Data":"0189dd5a37475c527971f77cb6be9b96972270e421ad78454761bfd75d715694"}
Oct 30 14:01:58.863012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:01:58.862853    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0189dd5a37475c527971f77cb6be9b96972270e421ad78454761bfd75d715694"
Oct 30 14:02:00.653134 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:00.653090    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"]
Oct 30 14:02:00.653613 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:00.653351    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy" containerID="cri-o://533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61" gracePeriod=30
Oct 30 14:02:04.755723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:04.755686    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"]
Oct 30 14:02:04.790070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:04.790022    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"]
Oct 30 14:02:05.935681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.935581    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 14:02:05.943971 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.943941    4291 generic.go:356] "Generic (PLEG): container finished" podID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerID="533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61" exitCode=0
Oct 30 14:02:05.944123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.943996    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"
Oct 30 14:02:05.944123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.944025    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" event={"ID":"31a6bbcd-4039-492d-8c6b-079648353ef2","Type":"ContainerDied","Data":"533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61"}
Oct 30 14:02:05.944123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.944082    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt" event={"ID":"31a6bbcd-4039-492d-8c6b-079648353ef2","Type":"ContainerDied","Data":"9a9b4c361da03639d8035673aad85c480daae9c1918cc63be1ab6b1c726e64b8"}
Oct 30 14:02:05.944123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.944115    4291 scope.go:122] "RemoveContainer" containerID="533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61"
Oct 30 14:02:05.955621 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.955599    4291 scope.go:122] "RemoveContainer" containerID="ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa"
Oct 30 14:02:05.973203 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973171    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-certs\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973261    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-token\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-token\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973310    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-socket\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973372    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-envoy\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973421    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-kube-api-access-s2wjx\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-kube-api-access-s2wjx\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973475    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-credential-socket\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973515    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-data\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973535    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-certs" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:05.973601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973550    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-podinfo\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973895 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973606    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31a6bbcd-4039-492d-8c6b-079648353ef2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a6bbcd-4039-492d-8c6b-079648353ef2-istiod-ca-cert\") pod \"31a6bbcd-4039-492d-8c6b-079648353ef2\" (UID: \"31a6bbcd-4039-492d-8c6b-079648353ef2\") "
Oct 30 14:02:05.973983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.973957    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-credential-socket" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:05.974167 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.974116    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-socket" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:05.974368 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.974294    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31a6bbcd-4039-492d-8c6b-079648353ef2-istiod-ca-cert" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:02:05.974487 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.974413    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:05.974487 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.974436    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:05.974705 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.974529    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-data" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:05.976686 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.976608    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-token" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:05.976810 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.976733    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-envoy" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:05.976882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.976830    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-kube-api-access-s2wjx" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "kube-api-access-s2wjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:05.976947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.976912    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-podinfo" pod "31a6bbcd-4039-492d-8c6b-079648353ef2" (UID: "31a6bbcd-4039-492d-8c6b-079648353ef2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:02:05.987520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.987495    4291 scope.go:122] "RemoveContainer" containerID="533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61"
Oct 30 14:02:05.987911 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:02:05.987889    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61\": container with ID starting with 533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61 not found: ID does not exist" containerID="533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61"
Oct 30 14:02:05.988007 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.987920    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61"} err="failed to get container status \"533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61\": rpc error: code = NotFound desc = could not find container \"533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61\": container with ID starting with 533509a5e4acfdacc57c34eb1ed5561c22959b9d9a71b782757bae4e0a887c61 not found: ID does not exist"
Oct 30 14:02:05.988007 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.987940    4291 scope.go:122] "RemoveContainer" containerID="ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa"
Oct 30 14:02:05.988256 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:02:05.988232    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa\": container with ID starting with ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa not found: ID does not exist" containerID="ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa"
Oct 30 14:02:05.988338 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:05.988265    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa"} err="failed to get container status \"ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa\": rpc error: code = NotFound desc = could not find container \"ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa\": container with ID starting with ec010415201021592c17cca815e2d4b98f54cbea889a4d2f4bef5070515583fa not found: ID does not exist"
Oct 30 14:02:06.075529 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.075494    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:06.075529 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.075525    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:06.075529 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.075534    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:06.075801 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.075543    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s2wjx\" (UniqueName: \"kubernetes.io/projected/31a6bbcd-4039-492d-8c6b-079648353ef2-kube-api-access-s2wjx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:06.075801 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.075553    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:06.075801 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.075562    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31a6bbcd-4039-492d-8c6b-079648353ef2-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:06.075801 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.075569    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31a6bbcd-4039-492d-8c6b-079648353ef2-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:06.291254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.291210    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"]
Oct 30 14:02:06.302460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:06.302427    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-09014b-openshift-default-bdbd9f58f-rcqvt"]
Oct 30 14:02:07.881883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:07.881841    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" path="/var/lib/kubelet/pods/31a6bbcd-4039-492d-8c6b-079648353ef2/volumes"
Oct 30 14:02:08.212423 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:02:08.212331    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:02:09.756287 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:09.756247    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz" podUID="d703d9f4-f5c2-4266-a584-83de96a71a23" containerName="payload-processing" containerID="cri-o://ed59dfb20f6a2fb6d913125796532f60d2acd9e2151c6655a8edddaccee3fc97" gracePeriod=25
Oct 30 14:02:09.790625 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:09.790586    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" podUID="22e89ec9-a212-4721-8724-77dea9a5bab7" containerName="payload-pre-processing" containerID="cri-o://4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735" gracePeriod=25
Oct 30 14:02:09.977858 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:09.977825    4291 generic.go:356] "Generic (PLEG): container finished" podID="d703d9f4-f5c2-4266-a584-83de96a71a23" containerID="ed59dfb20f6a2fb6d913125796532f60d2acd9e2151c6655a8edddaccee3fc97" exitCode=0
Oct 30 14:02:09.978038 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:09.977905    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz" event={"ID":"d703d9f4-f5c2-4266-a584-83de96a71a23","Type":"ContainerDied","Data":"ed59dfb20f6a2fb6d913125796532f60d2acd9e2151c6655a8edddaccee3fc97"}
Oct 30 14:02:10.048571 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.048545    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 14:02:10.125142 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.125104    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d703d9f4-f5c2-4266-a584-83de96a71a23-kube-api-access-mmmnb\" (UniqueName: \"kubernetes.io/projected/d703d9f4-f5c2-4266-a584-83de96a71a23-kube-api-access-mmmnb\") pod \"d703d9f4-f5c2-4266-a584-83de96a71a23\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") "
Oct 30 14:02:10.125357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.125180    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d703d9f4-f5c2-4266-a584-83de96a71a23-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d703d9f4-f5c2-4266-a584-83de96a71a23-plugins-config-volume\") pod \"d703d9f4-f5c2-4266-a584-83de96a71a23\" (UID: \"d703d9f4-f5c2-4266-a584-83de96a71a23\") "
Oct 30 14:02:10.125611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.125585    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d703d9f4-f5c2-4266-a584-83de96a71a23-plugins-config-volume" pod "d703d9f4-f5c2-4266-a584-83de96a71a23" (UID: "d703d9f4-f5c2-4266-a584-83de96a71a23"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:02:10.126011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.125984    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d703d9f4-f5c2-4266-a584-83de96a71a23-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:10.128119 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.128064    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d703d9f4-f5c2-4266-a584-83de96a71a23-kube-api-access-mmmnb" pod "d703d9f4-f5c2-4266-a584-83de96a71a23" (UID: "d703d9f4-f5c2-4266-a584-83de96a71a23"). InnerVolumeSpecName "kube-api-access-mmmnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:10.227727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.227695    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmmnb\" (UniqueName: \"kubernetes.io/projected/d703d9f4-f5c2-4266-a584-83de96a71a23-kube-api-access-mmmnb\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:10.501787 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.501747    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" podUID="22e89ec9-a212-4721-8724-77dea9a5bab7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:02:10.987190 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.987153    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz" event={"ID":"d703d9f4-f5c2-4266-a584-83de96a71a23","Type":"ContainerDied","Data":"7efc91f39bdaeb4e19c78e3954d2aae9eb5dfa05f61e2f0e280976c557f0add9"}
Oct 30 14:02:10.987678 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.987200    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"
Oct 30 14:02:10.987678 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:10.987259    4291 scope.go:122] "RemoveContainer" containerID="ed59dfb20f6a2fb6d913125796532f60d2acd9e2151c6655a8edddaccee3fc97"
Oct 30 14:02:11.039238 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:11.039197    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"]
Oct 30 14:02:11.050477 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:11.050444    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-319179-775886b666-7vtvz"]
Oct 30 14:02:11.881840 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:11.881807    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d703d9f4-f5c2-4266-a584-83de96a71a23" path="/var/lib/kubelet/pods/d703d9f4-f5c2-4266-a584-83de96a71a23/volumes"
Oct 30 14:02:15.036129 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:15.036088    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"]
Oct 30 14:02:15.047441 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:15.047411    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fb1a3074-pw9w5"]
Oct 30 14:02:15.881130 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:15.881091    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a039de63-693b-43c6-bfb5-f7ab327d4be1" path="/var/lib/kubelet/pods/a039de63-693b-43c6-bfb5-f7ab327d4be1/volumes"
Oct 30 14:02:16.352854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:16.352816    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:02:20.501339 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:20.501296    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" podUID="22e89ec9-a212-4721-8724-77dea9a5bab7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:02:27.031508 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:27.031415    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"]
Oct 30 14:02:27.043508 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:27.043472    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-776c2587-rb8sr"]
Oct 30 14:02:27.880242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:27.880204    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d63e9d83-6131-421c-9588-b68fd33e9128" path="/var/lib/kubelet/pods/d63e9d83-6131-421c-9588-b68fd33e9128/volumes"
Oct 30 14:02:30.062130 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.062067    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 14:02:30.138870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.138837    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22e89ec9-a212-4721-8724-77dea9a5bab7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e89ec9-a212-4721-8724-77dea9a5bab7-plugins-config-volume\") pod \"22e89ec9-a212-4721-8724-77dea9a5bab7\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") "
Oct 30 14:02:30.139159 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.139063    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22e89ec9-a212-4721-8724-77dea9a5bab7-kube-api-access-5z6nn\" (UniqueName: \"kubernetes.io/projected/22e89ec9-a212-4721-8724-77dea9a5bab7-kube-api-access-5z6nn\") pod \"22e89ec9-a212-4721-8724-77dea9a5bab7\" (UID: \"22e89ec9-a212-4721-8724-77dea9a5bab7\") "
Oct 30 14:02:30.139344 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.139314    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22e89ec9-a212-4721-8724-77dea9a5bab7-plugins-config-volume" pod "22e89ec9-a212-4721-8724-77dea9a5bab7" (UID: "22e89ec9-a212-4721-8724-77dea9a5bab7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:02:30.139747 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.139724    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22e89ec9-a212-4721-8724-77dea9a5bab7-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:30.141418 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.141387    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22e89ec9-a212-4721-8724-77dea9a5bab7-kube-api-access-5z6nn" pod "22e89ec9-a212-4721-8724-77dea9a5bab7" (UID: "22e89ec9-a212-4721-8724-77dea9a5bab7"). InnerVolumeSpecName "kube-api-access-5z6nn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:30.159847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.159822    4291 generic.go:356] "Generic (PLEG): container finished" podID="22e89ec9-a212-4721-8724-77dea9a5bab7" containerID="4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735" exitCode=0
Oct 30 14:02:30.159973 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.159877    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" event={"ID":"22e89ec9-a212-4721-8724-77dea9a5bab7","Type":"ContainerDied","Data":"4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735"}
Oct 30 14:02:30.159973 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.159925    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw" event={"ID":"22e89ec9-a212-4721-8724-77dea9a5bab7","Type":"ContainerDied","Data":"5712ac42b895a609e0fbd1b3deeb163ae8c04be6ed5fecc53a5b52015b8baba5"}
Oct 30 14:02:30.159973 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.159947    4291 scope.go:122] "RemoveContainer" containerID="4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735"
Oct 30 14:02:30.159973 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.159880    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"
Oct 30 14:02:30.175323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.175290    4291 scope.go:122] "RemoveContainer" containerID="4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735"
Oct 30 14:02:30.175915 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:02:30.175878    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735\": container with ID starting with 4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735 not found: ID does not exist" containerID="4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735"
Oct 30 14:02:30.176126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.175923    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735"} err="failed to get container status \"4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735\": rpc error: code = NotFound desc = could not find container \"4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735\": container with ID starting with 4d8bdee8f4b8ed9c27e1f496ac07657fd4e2a2da9c3e67e58b5a1a14439e1735 not found: ID does not exist"
Oct 30 14:02:30.230842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.230480    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"]
Oct 30 14:02:30.234298 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.234269    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-319179-645444b5f5-c5blw"]
Oct 30 14:02:30.241432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:30.241407    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z6nn\" (UniqueName: \"kubernetes.io/projected/22e89ec9-a212-4721-8724-77dea9a5bab7-kube-api-access-5z6nn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:31.879767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:31.879731    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22e89ec9-a212-4721-8724-77dea9a5bab7" path="/var/lib/kubelet/pods/22e89ec9-a212-4721-8724-77dea9a5bab7/volumes"
Oct 30 14:02:38.213399 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:02:38.213360    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:02:40.226541 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:40.226507    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"]
Oct 30 14:02:40.230800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:40.230771    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-319179-vb464"]
Oct 30 14:02:41.879701 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:41.879658    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc977e5-25f8-4ba3-8280-b751da1d05f8" path="/var/lib/kubelet/pods/ccc977e5-25f8-4ba3-8280-b751da1d05f8/volumes"
Oct 30 14:02:42.350822 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.350783    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"]
Oct 30 14:02:42.351679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351662    4291 cpu_manager.go:414] "containerMap: removing container" podUID="22e89ec9-a212-4721-8724-77dea9a5bab7" containerName="payload-pre-processing"
Oct 30 14:02:42.351733 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351682    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22e89ec9-a212-4721-8724-77dea9a5bab7" containerName="payload-pre-processing"
Oct 30 14:02:42.351781 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351733    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ccc977e5-25f8-4ba3-8280-b751da1d05f8" containerName="revoke-keys"
Oct 30 14:02:42.351781 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351742    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc977e5-25f8-4ba3-8280-b751da1d05f8" containerName="revoke-keys"
Oct 30 14:02:42.351781 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351771    4291 cpu_manager.go:414] "containerMap: removing container" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy"
Oct 30 14:02:42.351781 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351778    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy"
Oct 30 14:02:42.351924 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351805    4291 cpu_manager.go:414] "containerMap: removing container" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy"
Oct 30 14:02:42.351924 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351813    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy"
Oct 30 14:02:42.351924 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351841    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d703d9f4-f5c2-4266-a584-83de96a71a23" containerName="payload-processing"
Oct 30 14:02:42.351924 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.351848    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d703d9f4-f5c2-4266-a584-83de96a71a23" containerName="payload-processing"
Oct 30 14:02:42.352056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.352003    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy"
Oct 30 14:02:42.352056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.352016    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d703d9f4-f5c2-4266-a584-83de96a71a23" containerName="payload-processing"
Oct 30 14:02:42.352056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.352026    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="22e89ec9-a212-4721-8724-77dea9a5bab7" containerName="payload-pre-processing"
Oct 30 14:02:42.352056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.352043    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc977e5-25f8-4ba3-8280-b751da1d05f8" containerName="revoke-keys"
Oct 30 14:02:42.357215 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.357190    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.370507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.370480    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"]
Oct 30 14:02:42.378560 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.378470    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:02:42.378744 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.378580    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:02:42.378744 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.378584    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:02:42.383418 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.383394    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:02:42.436578 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.436541    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40faca54-8328-4608-9d45-5b13343021fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.436809 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.436791    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40faca54-8328-4608-9d45-5b13343021fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.436866 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.436823    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z8v8\" (UniqueName: \"kubernetes.io/projected/40faca54-8328-4608-9d45-5b13343021fd-kube-api-access-4z8v8\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.537955 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.537915    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40faca54-8328-4608-9d45-5b13343021fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.538175 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.537965    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z8v8\" (UniqueName: \"kubernetes.io/projected/40faca54-8328-4608-9d45-5b13343021fd-kube-api-access-4z8v8\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.538175 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.538063    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40faca54-8328-4608-9d45-5b13343021fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.538825 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.538798    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40faca54-8328-4608-9d45-5b13343021fd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.540487 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.540468    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40faca54-8328-4608-9d45-5b13343021fd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.557301 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.557269    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z8v8\" (UniqueName: \"kubernetes.io/projected/40faca54-8328-4608-9d45-5b13343021fd-kube-api-access-4z8v8\") pod \"maas-api-revoke-sub-8f8858b0-6z4vl\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.673123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.673034    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:42.975997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:42.975967    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"]
Oct 30 14:02:42.977174 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:02:42.977147    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40faca54_8328_4608_9d45_5b13343021fd.slice/crio-499ced8d0df488f930174cd2e575cbf5de06bd8983b098d25400bbdb0288f2c0 WatchSource:0}: Error finding container 499ced8d0df488f930174cd2e575cbf5de06bd8983b098d25400bbdb0288f2c0: Status 404 returned error can't find the container with id 499ced8d0df488f930174cd2e575cbf5de06bd8983b098d25400bbdb0288f2c0
Oct 30 14:02:43.259513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:43.259480    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl" event={"ID":"40faca54-8328-4608-9d45-5b13343021fd","Type":"ContainerStarted","Data":"499ced8d0df488f930174cd2e575cbf5de06bd8983b098d25400bbdb0288f2c0"}
Oct 30 14:02:44.268076 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:44.267988    4291 generic.go:356] "Generic (PLEG): container finished" podID="40faca54-8328-4608-9d45-5b13343021fd" containerID="87f6b0cc8df45121b52c7de62e216aa7589cb94de24dae75497129f7b0cf9222" exitCode=0
Oct 30 14:02:44.268443 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:44.268077    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl" event={"ID":"40faca54-8328-4608-9d45-5b13343021fd","Type":"ContainerDied","Data":"87f6b0cc8df45121b52c7de62e216aa7589cb94de24dae75497129f7b0cf9222"}
Oct 30 14:02:45.436017 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.435987    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:45.577619 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.577528    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40faca54-8328-4608-9d45-5b13343021fd-kube-api-access-4z8v8\" (UniqueName: \"kubernetes.io/projected/40faca54-8328-4608-9d45-5b13343021fd-kube-api-access-4z8v8\") pod \"40faca54-8328-4608-9d45-5b13343021fd\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") "
Oct 30 14:02:45.577846 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.577753    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40faca54-8328-4608-9d45-5b13343021fd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40faca54-8328-4608-9d45-5b13343021fd-maas-api-xks-ca\") pod \"40faca54-8328-4608-9d45-5b13343021fd\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") "
Oct 30 14:02:45.577919 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.577858    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40faca54-8328-4608-9d45-5b13343021fd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40faca54-8328-4608-9d45-5b13343021fd-maas-api-openshift-ca\") pod \"40faca54-8328-4608-9d45-5b13343021fd\" (UID: \"40faca54-8328-4608-9d45-5b13343021fd\") "
Oct 30 14:02:45.578285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.578255    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40faca54-8328-4608-9d45-5b13343021fd-maas-api-openshift-ca" pod "40faca54-8328-4608-9d45-5b13343021fd" (UID: "40faca54-8328-4608-9d45-5b13343021fd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:02:45.578423 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.578405    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/40faca54-8328-4608-9d45-5b13343021fd-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:45.579986 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.579967    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40faca54-8328-4608-9d45-5b13343021fd-maas-api-xks-ca" pod "40faca54-8328-4608-9d45-5b13343021fd" (UID: "40faca54-8328-4608-9d45-5b13343021fd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:02:45.580152 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.580133    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40faca54-8328-4608-9d45-5b13343021fd-kube-api-access-4z8v8" pod "40faca54-8328-4608-9d45-5b13343021fd" (UID: "40faca54-8328-4608-9d45-5b13343021fd"). InnerVolumeSpecName "kube-api-access-4z8v8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:45.680251 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.680148    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/40faca54-8328-4608-9d45-5b13343021fd-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:45.680251 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:45.680174    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z8v8\" (UniqueName: \"kubernetes.io/projected/40faca54-8328-4608-9d45-5b13343021fd-kube-api-access-4z8v8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:46.282877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:46.282801    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"
Oct 30 14:02:46.283035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:46.282799    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl" event={"ID":"40faca54-8328-4608-9d45-5b13343021fd","Type":"ContainerDied","Data":"499ced8d0df488f930174cd2e575cbf5de06bd8983b098d25400bbdb0288f2c0"}
Oct 30 14:02:46.283035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:46.282924    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="499ced8d0df488f930174cd2e575cbf5de06bd8983b098d25400bbdb0288f2c0"
Oct 30 14:02:48.337357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.337319    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"]
Oct 30 14:02:48.337795 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.337599    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" containerID="cri-o://b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2" gracePeriod=30
Oct 30 14:02:48.587849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.587738    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"]
Oct 30 14:02:48.588505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.588475    4291 cpu_manager.go:414] "containerMap: removing container" podUID="40faca54-8328-4608-9d45-5b13343021fd" containerName="revoke-keys"
Oct 30 14:02:48.588505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.588497    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40faca54-8328-4608-9d45-5b13343021fd" containerName="revoke-keys"
Oct 30 14:02:48.588751 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.588609    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="40faca54-8328-4608-9d45-5b13343021fd" containerName="revoke-keys"
Oct 30 14:02:48.588751 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.588624    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="31a6bbcd-4039-492d-8c6b-079648353ef2" containerName="istio-proxy"
Oct 30 14:02:48.592997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.592971    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.616769 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.616732    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"]
Oct 30 14:02:48.626305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.626255    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"]
Oct 30 14:02:48.632072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.631895    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.668385 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.668352    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"]
Oct 30 14:02:48.737750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.737718    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.737750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.737758    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.738051 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.737882    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.738051 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.737958    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.738173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738078    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.738173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738121    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.738281 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738216    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.738348 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738278    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-kube-api-access-szrhx\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.738439 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738416    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.738484 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738456    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.738597 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738575    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.738753 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738660    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgtlt\" (UniqueName: \"kubernetes.io/projected/57ce3df4-b78d-4de7-aecc-5364df414c55-kube-api-access-zgtlt\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.738753 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738705    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.738910 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738807    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.738910 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738874    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ce3df4-b78d-4de7-aecc-5364df414c55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.739043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738931    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.739043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.738959    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.739173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.739088    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.840511 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840423    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.840690 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840601    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.840739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840697    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.840739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840725    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.840827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840751    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-kube-api-access-szrhx\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.840827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840801    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.840921 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840830    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.840990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840944    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.841052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.840998    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgtlt\" (UniqueName: \"kubernetes.io/projected/57ce3df4-b78d-4de7-aecc-5364df414c55-kube-api-access-zgtlt\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841035    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841170 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841105    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.841170 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841142    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ce3df4-b78d-4de7-aecc-5364df414c55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841188    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841214    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.841285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841282    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.841458 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841305    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841458 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841329    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.841458 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841362    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841630 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841577    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841761 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841736    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.841840 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841791    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841913 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841882    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-workload-certs\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.841976 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.841932    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.842042 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.842014    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.842318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.842292    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-credential-socket\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.842504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.842474    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-data\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.842627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.842546    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57ce3df4-b78d-4de7-aecc-5364df414c55-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.842743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.842685    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.843237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.843221    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.843512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.843491    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.844071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.844051    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-envoy\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.844174 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.844119    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-podinfo\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.875029 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.875002    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.875255 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.875231    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57ce3df4-b78d-4de7-aecc-5364df414c55-istio-token\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.875587 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.875568    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-kube-api-access-szrhx\") pod \"maas-default-gateway-openshift-default-67bf986f99-qwt8c\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.880655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.880621    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgtlt\" (UniqueName: \"kubernetes.io/projected/57ce3df4-b78d-4de7-aecc-5364df414c55-kube-api-access-zgtlt\") pod \"maas-default-gateway-openshift-default-67bf986f99-b5924\" (UID: \"57ce3df4-b78d-4de7-aecc-5364df414c55\") " pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:48.910932 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.910903    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:48.964799 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:48.964765    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:49.270538 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.270501    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"]
Oct 30 14:02:49.271405 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:02:49.271376    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59c27364_1fb1_41b1_aee1_5491e06b4f6f.slice/crio-1cef2097d973ded8456621d83a4c204d8155c46ad8b6e11c57263a273d17b071 WatchSource:0}: Error finding container 1cef2097d973ded8456621d83a4c204d8155c46ad8b6e11c57263a273d17b071: Status 404 returned error can't find the container with id 1cef2097d973ded8456621d83a4c204d8155c46ad8b6e11c57263a273d17b071
Oct 30 14:02:49.277064 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.277022    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.277190 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.277108    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.277190 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.277141    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.312674 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.312647    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" event={"ID":"59c27364-1fb1-41b1-aee1-5491e06b4f6f","Type":"ContainerStarted","Data":"1cef2097d973ded8456621d83a4c204d8155c46ad8b6e11c57263a273d17b071"}
Oct 30 14:02:49.346939 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.346911    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"]
Oct 30 14:02:49.348325 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:02:49.348300    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57ce3df4_b78d_4de7_aecc_5364df414c55.slice/crio-be0d68591877cfee78ce459c346b57d3aa2d883aa79f513efeb443e986f1c7c0 WatchSource:0}: Error finding container be0d68591877cfee78ce459c346b57d3aa2d883aa79f513efeb443e986f1c7c0: Status 404 returned error can't find the container with id be0d68591877cfee78ce459c346b57d3aa2d883aa79f513efeb443e986f1c7c0
Oct 30 14:02:49.351282 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.351250    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.351366 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.351317    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:49.351366 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:49.351347    4291 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Oct 30 14:02:50.322785 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.322743    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" event={"ID":"57ce3df4-b78d-4de7-aecc-5364df414c55","Type":"ContainerStarted","Data":"645dd5e7a67dca05f02917f4206075f1ad803341896cbf1318096a5906eca5f9"}
Oct 30 14:02:50.322964 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.322808    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" event={"ID":"57ce3df4-b78d-4de7-aecc-5364df414c55","Type":"ContainerStarted","Data":"be0d68591877cfee78ce459c346b57d3aa2d883aa79f513efeb443e986f1c7c0"}
Oct 30 14:02:50.324714 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.324683    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" event={"ID":"59c27364-1fb1-41b1-aee1-5491e06b4f6f","Type":"ContainerStarted","Data":"287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e"}
Oct 30 14:02:50.403660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.401317    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podStartSLOduration=2.401295781 podStartE2EDuration="2.401295781s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:50.383025985 +0000 UTC m=+3580.643264230" watchObservedRunningTime="2026-10-30 14:02:50.401295781 +0000 UTC m=+3580.661534015"
Oct 30 14:02:50.438163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.438110    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podStartSLOduration=2.438094814 podStartE2EDuration="2.438094814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:50.43395043 +0000 UTC m=+3580.694188664" watchObservedRunningTime="2026-10-30 14:02:50.438094814 +0000 UTC m=+3580.698333048"
Oct 30 14:02:50.911198 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.911161    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:50.913198 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.913172    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qwt8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body=
Oct 30 14:02:50.913391 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.913367    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused"
Oct 30 14:02:50.978661 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.966000    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:51.000854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.983797    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b5924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Oct 30 14:02:51.000854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:50.983878    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podUID="57ce3df4-b78d-4de7-aecc-5364df414c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Oct 30 14:02:51.911341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:51.911305    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qwt8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body=
Oct 30 14:02:51.911909 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:51.911417    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused"
Oct 30 14:02:51.970660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:51.970076    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b5924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Oct 30 14:02:51.970660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:51.970172    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podUID="57ce3df4-b78d-4de7-aecc-5364df414c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Oct 30 14:02:52.822918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.822874    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"]
Oct 30 14:02:52.827245 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.827227    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:52.848549 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.848520    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:02:52.848843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.848823    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:02:52.891660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.887855    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:02:52.891660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.887954    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:02:52.912528 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.912485    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qwt8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body=
Oct 30 14:02:52.913066 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.912559    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused"
Oct 30 14:02:52.914750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.914714    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"]
Oct 30 14:02:52.933767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.931724    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lkpj\" (UniqueName: \"kubernetes.io/projected/78b116b8-d774-4c7b-a178-0e67c91904f4-kube-api-access-6lkpj\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:52.933767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.931761    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:52.933767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.931938    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:52.976661 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.973364    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b5924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Oct 30 14:02:52.976661 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:52.973420    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podUID="57ce3df4-b78d-4de7-aecc-5364df414c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Oct 30 14:02:53.033631 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.033230    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:53.033631 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.033338    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lkpj\" (UniqueName: \"kubernetes.io/projected/78b116b8-d774-4c7b-a178-0e67c91904f4-kube-api-access-6lkpj\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:53.033631 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.033358    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:53.034125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.034105    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:53.035748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.035729    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:53.113462 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.113434    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lkpj\" (UniqueName: \"kubernetes.io/projected/78b116b8-d774-4c7b-a178-0e67c91904f4-kube-api-access-6lkpj\") pod \"maas-api-revoke-sub-ce33b1ad-fnn8d\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:53.142297 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.142264    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:53.685930 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.680476    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"]
Oct 30 14:02:53.701769 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.699825    4291 patch_prober.go:29] interesting pod/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused" start-of-body=
Oct 30 14:02:53.701769 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.699910    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.39:15021/healthz/ready\": dial tcp 10.133.1.39:15021: connect: connection refused"
Oct 30 14:02:53.912649 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.912117    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qwt8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body=
Oct 30 14:02:53.912649 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.912180    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused"
Oct 30 14:02:53.954892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.950819    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"]
Oct 30 14:02:53.974883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.969093    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b5924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Oct 30 14:02:53.974883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:53.969157    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podUID="57ce3df4-b78d-4de7-aecc-5364df414c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Oct 30 14:02:54.012659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.011876    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"]
Oct 30 14:02:54.012659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.012010    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.110072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.110042    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 14:02:54.113753 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.113727    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.113888 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.113876    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-784g4\" (UniqueName: \"kubernetes.io/projected/5be8050e-8c6d-4e21-831d-602335f316c6-kube-api-access-784g4\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.113985 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.113974    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.215626 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.215595    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-credential-socket\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.215892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.215868    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-envoy\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.216139 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.216121    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-certs\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.216280 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.216264    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/769a0ac3-b411-46d3-894c-0855bbeab4a6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a0ac3-b411-46d3-894c-0855bbeab4a6-istiod-ca-cert\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.216427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.216411    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-socket\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.216678 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.216659    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-data\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.216843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.216827    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-podinfo\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.217070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.217053    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-token\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-token\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.217201 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.217186    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-kube-api-access-nxdcr\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-kube-api-access-nxdcr\") pod \"769a0ac3-b411-46d3-894c-0855bbeab4a6\" (UID: \"769a0ac3-b411-46d3-894c-0855bbeab4a6\") "
Oct 30 14:02:54.217848 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.217824    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.217990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.217973    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-784g4\" (UniqueName: \"kubernetes.io/projected/5be8050e-8c6d-4e21-831d-602335f316c6-kube-api-access-784g4\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.218128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.218114    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.218764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.216908    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-certs" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:54.218865 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.217099    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-credential-socket" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:54.219435 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.219408    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-data" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:54.219554 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.219427    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.219709 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.219688    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/769a0ac3-b411-46d3-894c-0855bbeab4a6-istiod-ca-cert" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:02:54.219887 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.219868    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-socket" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:54.222903 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.222877    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-envoy" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:02:54.223526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.223501    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.223998 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.223970    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-kube-api-access-nxdcr" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "kube-api-access-nxdcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:54.225768 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.225735    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-podinfo" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:02:54.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.225761    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-token" pod "769a0ac3-b411-46d3-894c-0855bbeab4a6" (UID: "769a0ac3-b411-46d3-894c-0855bbeab4a6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:54.241706 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.241677    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-784g4\" (UniqueName: \"kubernetes.io/projected/5be8050e-8c6d-4e21-831d-602335f316c6-kube-api-access-784g4\") pod \"maas-api-revoke-sub-2773d612-n7hsf\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.319513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319465    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319506    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319517    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319526    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319535    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxdcr\" (UniqueName: \"kubernetes.io/projected/769a0ac3-b411-46d3-894c-0855bbeab4a6-kube-api-access-nxdcr\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319544    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319553    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319561    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/769a0ac3-b411-46d3-894c-0855bbeab4a6-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.319837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.319570    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/769a0ac3-b411-46d3-894c-0855bbeab4a6-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:54.408302 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.408259    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:54.444918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.444884    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d" event={"ID":"78b116b8-d774-4c7b-a178-0e67c91904f4","Type":"ContainerStarted","Data":"e198717ae8a55340c85f344d99ae2a61f1b433b1653b180b412552bad355c4e9"}
Oct 30 14:02:54.445126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.444948    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d" event={"ID":"78b116b8-d774-4c7b-a178-0e67c91904f4","Type":"ContainerStarted","Data":"f8e104d22cf9bcb39687e73d3cf829304e06c48f0c60dbe08763352f11944c97"}
Oct 30 14:02:54.447376 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.447346    4291 generic.go:356] "Generic (PLEG): container finished" podID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerID="b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2" exitCode=0
Oct 30 14:02:54.447526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.447390    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" event={"ID":"769a0ac3-b411-46d3-894c-0855bbeab4a6","Type":"ContainerDied","Data":"b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2"}
Oct 30 14:02:54.447526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.447402    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"
Oct 30 14:02:54.447526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.447433    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v" event={"ID":"769a0ac3-b411-46d3-894c-0855bbeab4a6","Type":"ContainerDied","Data":"6b1f7ec93bbbf10e275f48d701d177e371b6663d06ea3235bad62919637b7354"}
Oct 30 14:02:54.447526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.447463    4291 scope.go:122] "RemoveContainer" containerID="b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2"
Oct 30 14:02:54.513595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.513389    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d" podStartSLOduration=2.513372074 podStartE2EDuration="2.513372074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:54.511456212 +0000 UTC m=+3584.771694446" watchObservedRunningTime="2026-10-30 14:02:54.513372074 +0000 UTC m=+3584.773610308"
Oct 30 14:02:54.528470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.528439    4291 scope.go:122] "RemoveContainer" containerID="264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d"
Oct 30 14:02:54.612382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.612337    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"]
Oct 30 14:02:54.619907 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.619873    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-319179-openshift-default-568ff76ccc-9545v"]
Oct 30 14:02:54.638507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.638472    4291 scope.go:122] "RemoveContainer" containerID="b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2"
Oct 30 14:02:54.638883 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:02:54.638859    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2\": container with ID starting with b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2 not found: ID does not exist" containerID="b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2"
Oct 30 14:02:54.638997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.638895    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2"} err="failed to get container status \"b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2\": rpc error: code = NotFound desc = could not find container \"b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2\": container with ID starting with b117ff0ec72f899dd671d447e8cc8608605b8ddbcc73dcbc804b7f4f2db373f2 not found: ID does not exist"
Oct 30 14:02:54.638997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.638923    4291 scope.go:122] "RemoveContainer" containerID="264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d"
Oct 30 14:02:54.639223 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:02:54.639201    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d\": container with ID starting with 264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d not found: ID does not exist" containerID="264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d"
Oct 30 14:02:54.639296 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.639228    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d"} err="failed to get container status \"264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d\": rpc error: code = NotFound desc = could not find container \"264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d\": container with ID starting with 264dc1e74e465879d2f6932456a6a861f919375ae0aea78c809204be89f8ac9d not found: ID does not exist"
Oct 30 14:02:54.808014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.807887    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"]
Oct 30 14:02:54.812069 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:02:54.812034    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be8050e_8c6d_4e21_831d_602335f316c6.slice/crio-4a86e2856d33b611c5f9682f7498ec7c2ff914754c9a818ae374be1730d1a678 WatchSource:0}: Error finding container 4a86e2856d33b611c5f9682f7498ec7c2ff914754c9a818ae374be1730d1a678: Status 404 returned error can't find the container with id 4a86e2856d33b611c5f9682f7498ec7c2ff914754c9a818ae374be1730d1a678
Oct 30 14:02:54.914219 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.912371    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qwt8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body=
Oct 30 14:02:54.914219 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.912422    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused"
Oct 30 14:02:55.000277 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.999796    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b5924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Oct 30 14:02:55.000277 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:54.999870    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podUID="57ce3df4-b78d-4de7-aecc-5364df414c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Oct 30 14:02:55.512690 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.509839    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf" event={"ID":"5be8050e-8c6d-4e21-831d-602335f316c6","Type":"ContainerStarted","Data":"e7de6db1474ed169d74c6bfff7a985473b8593e98f0c5bce7837de86d952d1c8"}
Oct 30 14:02:55.512690 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.509934    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf" event={"ID":"5be8050e-8c6d-4e21-831d-602335f316c6","Type":"ContainerStarted","Data":"4a86e2856d33b611c5f9682f7498ec7c2ff914754c9a818ae374be1730d1a678"}
Oct 30 14:02:55.528721 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.515339    4291 generic.go:356] "Generic (PLEG): container finished" podID="78b116b8-d774-4c7b-a178-0e67c91904f4" containerID="e198717ae8a55340c85f344d99ae2a61f1b433b1653b180b412552bad355c4e9" exitCode=0
Oct 30 14:02:55.528721 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.515555    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d" event={"ID":"78b116b8-d774-4c7b-a178-0e67c91904f4","Type":"ContainerDied","Data":"e198717ae8a55340c85f344d99ae2a61f1b433b1653b180b412552bad355c4e9"}
Oct 30 14:02:55.592745 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.590234    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf" podStartSLOduration=2.590215351 podStartE2EDuration="2.590215351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:02:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:02:55.559050951 +0000 UTC m=+3585.819289188" watchObservedRunningTime="2026-10-30 14:02:55.590215351 +0000 UTC m=+3585.850453586"
Oct 30 14:02:55.884281 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.884237    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" path="/var/lib/kubelet/pods/769a0ac3-b411-46d3-894c-0855bbeab4a6/volumes"
Oct 30 14:02:55.912135 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.912094    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qwt8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body=
Oct 30 14:02:55.912311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.912166    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused"
Oct 30 14:02:55.965847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.965803    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b5924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Oct 30 14:02:55.966269 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:55.965875    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podUID="57ce3df4-b78d-4de7-aecc-5364df414c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Oct 30 14:02:56.526956 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.526917    4291 generic.go:356] "Generic (PLEG): container finished" podID="5be8050e-8c6d-4e21-831d-602335f316c6" containerID="e7de6db1474ed169d74c6bfff7a985473b8593e98f0c5bce7837de86d952d1c8" exitCode=0
Oct 30 14:02:56.527168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.526994    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf" event={"ID":"5be8050e-8c6d-4e21-831d-602335f316c6","Type":"ContainerDied","Data":"e7de6db1474ed169d74c6bfff7a985473b8593e98f0c5bce7837de86d952d1c8"}
Oct 30 14:02:56.770809 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.770769    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:56.911115 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.911074    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78b116b8-d774-4c7b-a178-0e67c91904f4-kube-api-access-6lkpj\" (UniqueName: \"kubernetes.io/projected/78b116b8-d774-4c7b-a178-0e67c91904f4-kube-api-access-6lkpj\") pod \"78b116b8-d774-4c7b-a178-0e67c91904f4\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") "
Oct 30 14:02:56.911326 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.911306    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-openshift-ca\") pod \"78b116b8-d774-4c7b-a178-0e67c91904f4\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") "
Oct 30 14:02:56.911393 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.911349    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-xks-ca\") pod \"78b116b8-d774-4c7b-a178-0e67c91904f4\" (UID: \"78b116b8-d774-4c7b-a178-0e67c91904f4\") "
Oct 30 14:02:56.911519 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.911492    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-qwt8c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused" start-of-body=
Oct 30 14:02:56.911604 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.911547    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.88:15021/healthz/ready\": dial tcp 10.133.1.88:15021: connect: connection refused"
Oct 30 14:02:56.911746 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.911724    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-openshift-ca" pod "78b116b8-d774-4c7b-a178-0e67c91904f4" (UID: "78b116b8-d774-4c7b-a178-0e67c91904f4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:02:56.912075 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.912056    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:56.913501 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.913478    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78b116b8-d774-4c7b-a178-0e67c91904f4-kube-api-access-6lkpj" pod "78b116b8-d774-4c7b-a178-0e67c91904f4" (UID: "78b116b8-d774-4c7b-a178-0e67c91904f4"). InnerVolumeSpecName "kube-api-access-6lkpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:56.914013 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.913987    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-xks-ca" pod "78b116b8-d774-4c7b-a178-0e67c91904f4" (UID: "78b116b8-d774-4c7b-a178-0e67c91904f4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:02:56.965350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.965315    4291 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-67bf986f99-b5924 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused" start-of-body=
Oct 30 14:02:56.965532 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:56.965377    4291 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924" podUID="57ce3df4-b78d-4de7-aecc-5364df414c55" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.89:15021/healthz/ready\": dial tcp 10.133.1.89:15021: connect: connection refused"
Oct 30 14:02:57.013838 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.013790    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/78b116b8-d774-4c7b-a178-0e67c91904f4-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:57.013838 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.013826    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lkpj\" (UniqueName: \"kubernetes.io/projected/78b116b8-d774-4c7b-a178-0e67c91904f4-kube-api-access-6lkpj\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:57.541762 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.541722    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"
Oct 30 14:02:57.543803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.543763    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d" event={"ID":"78b116b8-d774-4c7b-a178-0e67c91904f4","Type":"ContainerDied","Data":"f8e104d22cf9bcb39687e73d3cf829304e06c48f0c60dbe08763352f11944c97"}
Oct 30 14:02:57.543968 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.543850    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8e104d22cf9bcb39687e73d3cf829304e06c48f0c60dbe08763352f11944c97"
Oct 30 14:02:57.794538 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.794426    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:57.935167 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.935127    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-openshift-ca\") pod \"5be8050e-8c6d-4e21-831d-602335f316c6\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") "
Oct 30 14:02:57.935349 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.935274    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5be8050e-8c6d-4e21-831d-602335f316c6-kube-api-access-784g4\" (UniqueName: \"kubernetes.io/projected/5be8050e-8c6d-4e21-831d-602335f316c6-kube-api-access-784g4\") pod \"5be8050e-8c6d-4e21-831d-602335f316c6\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") "
Oct 30 14:02:57.935421 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.935349    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-xks-ca\") pod \"5be8050e-8c6d-4e21-831d-602335f316c6\" (UID: \"5be8050e-8c6d-4e21-831d-602335f316c6\") "
Oct 30 14:02:57.935586 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.935554    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-openshift-ca" pod "5be8050e-8c6d-4e21-831d-602335f316c6" (UID: "5be8050e-8c6d-4e21-831d-602335f316c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:02:57.936053 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.936023    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:57.938369 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.938337    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5be8050e-8c6d-4e21-831d-602335f316c6-kube-api-access-784g4" pod "5be8050e-8c6d-4e21-831d-602335f316c6" (UID: "5be8050e-8c6d-4e21-831d-602335f316c6"). InnerVolumeSpecName "kube-api-access-784g4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:02:57.939200 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:57.939171    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-xks-ca" pod "5be8050e-8c6d-4e21-831d-602335f316c6" (UID: "5be8050e-8c6d-4e21-831d-602335f316c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:02:58.038287 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.038226    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-784g4\" (UniqueName: \"kubernetes.io/projected/5be8050e-8c6d-4e21-831d-602335f316c6-kube-api-access-784g4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:58.038287 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.038273    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5be8050e-8c6d-4e21-831d-602335f316c6-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:02:58.264378 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.264343    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:58.265160 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.265048    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:58.266811 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.266788    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:02:58.338013 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.337983    4291 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:58.338321 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.338302    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:58.339307 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.339284    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-b5924"
Oct 30 14:02:58.550552 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.550463    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"
Oct 30 14:02:58.550552 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.550495    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf" event={"ID":"5be8050e-8c6d-4e21-831d-602335f316c6","Type":"ContainerDied","Data":"4a86e2856d33b611c5f9682f7498ec7c2ff914754c9a818ae374be1730d1a678"}
Oct 30 14:02:58.550552 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:02:58.550550    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a86e2856d33b611c5f9682f7498ec7c2ff914754c9a818ae374be1730d1a678"
Oct 30 14:03:05.038433 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:05.038391    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"]
Oct 30 14:03:05.048884 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:05.048856    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d6551a7f-df52j"]
Oct 30 14:03:05.880309 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:05.880276    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc54b30-fd04-4968-beb4-2b021bd92264" path="/var/lib/kubelet/pods/dbc54b30-fd04-4968-beb4-2b021bd92264/volumes"
Oct 30 14:03:08.211410 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:03:08.211371    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:03:11.489860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:11.489824    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:03:12.681077 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.681035    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"]
Oct 30 14:03:12.682254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.682170    4291 cpu_manager.go:414] "containerMap: removing container" podUID="78b116b8-d774-4c7b-a178-0e67c91904f4" containerName="revoke-keys"
Oct 30 14:03:12.682692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.682669    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78b116b8-d774-4c7b-a178-0e67c91904f4" containerName="revoke-keys"
Oct 30 14:03:12.682853 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.682841    4291 cpu_manager.go:414] "containerMap: removing container" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy"
Oct 30 14:03:12.682951 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.682938    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy"
Oct 30 14:03:12.683054 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683045    4291 cpu_manager.go:414] "containerMap: removing container" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy"
Oct 30 14:03:12.683136 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683125    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy"
Oct 30 14:03:12.683231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683218    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5be8050e-8c6d-4e21-831d-602335f316c6" containerName="revoke-keys"
Oct 30 14:03:12.683322 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683309    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5be8050e-8c6d-4e21-831d-602335f316c6" containerName="revoke-keys"
Oct 30 14:03:12.683597 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683581    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5be8050e-8c6d-4e21-831d-602335f316c6" containerName="revoke-keys"
Oct 30 14:03:12.683782 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683770    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="78b116b8-d774-4c7b-a178-0e67c91904f4" containerName="revoke-keys"
Oct 30 14:03:12.683891 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683880    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy"
Oct 30 14:03:12.683982 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.683972    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="769a0ac3-b411-46d3-894c-0855bbeab4a6" containerName="istio-proxy"
Oct 30 14:03:12.699657 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.695949    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.718668 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.710141    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"]
Oct 30 14:03:12.720826 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.720790    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:03:12.732958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.732923    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:03:12.733191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.733000    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:03:12.733191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.733054    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:03:12.758038 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.757999    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.758235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.758113    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fc28v\" (UniqueName: \"kubernetes.io/projected/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-kube-api-access-fc28v\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.758410 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.758390    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.859595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.859560    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.859789 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.859670    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.859789 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.859707    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fc28v\" (UniqueName: \"kubernetes.io/projected/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-kube-api-access-fc28v\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.860412 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.860385    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.862002 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.861977    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:12.879750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:12.879718    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fc28v\" (UniqueName: \"kubernetes.io/projected/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-kube-api-access-fc28v\") pod \"maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:13.030596 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:13.030446    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:13.539862 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:13.539821    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"]
Oct 30 14:03:13.688188 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:13.688147    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc" event={"ID":"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4","Type":"ContainerStarted","Data":"26c582260473ef32e8d87f80bde41ad5080b06e1c4b51ec3111494744362ff06"}
Oct 30 14:03:14.046387 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:14.046352    4291 scope.go:122] "RemoveContainer" containerID="a026d68e599449e59f0ab87c7826e84f121cf3f428d5f64662a49819416d215b"
Oct 30 14:03:14.749191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:14.749150    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc" event={"ID":"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4","Type":"ContainerStarted","Data":"d887cbdb095730d363a68010249dd75c4c28a49e837ad053ce32a6761436019d"}
Oct 30 14:03:15.793224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:15.793182    4291 generic.go:356] "Generic (PLEG): container finished" podID="ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" containerID="d887cbdb095730d363a68010249dd75c4c28a49e837ad053ce32a6761436019d" exitCode=0
Oct 30 14:03:15.794606 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:15.794190    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc" event={"ID":"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4","Type":"ContainerDied","Data":"d887cbdb095730d363a68010249dd75c4c28a49e837ad053ce32a6761436019d"}
Oct 30 14:03:16.982129 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:16.982103    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:17.062347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.062259    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-xks-ca\") pod \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") "
Oct 30 14:03:17.062347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.062332    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-kube-api-access-fc28v\" (UniqueName: \"kubernetes.io/projected/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-kube-api-access-fc28v\") pod \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") "
Oct 30 14:03:17.062605 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.062408    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-openshift-ca\") pod \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\" (UID: \"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4\") "
Oct 30 14:03:17.062885 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.062853    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-openshift-ca" pod "ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" (UID: "ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:03:17.063087 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.063069    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:03:17.065182 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.065152    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-kube-api-access-fc28v" pod "ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" (UID: "ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4"). InnerVolumeSpecName "kube-api-access-fc28v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:03:17.065281 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.065214    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-xks-ca" pod "ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" (UID: "ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:03:17.164488 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.164450    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:03:17.164488 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.164484    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fc28v\" (UniqueName: \"kubernetes.io/projected/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4-kube-api-access-fc28v\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:03:17.811212 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.811180    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"
Oct 30 14:03:17.811212 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.811201    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc" event={"ID":"ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4","Type":"ContainerDied","Data":"26c582260473ef32e8d87f80bde41ad5080b06e1c4b51ec3111494744362ff06"}
Oct 30 14:03:17.811446 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:17.811252    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26c582260473ef32e8d87f80bde41ad5080b06e1c4b51ec3111494744362ff06"
Oct 30 14:03:21.046070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:21.046008    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"]
Oct 30 14:03:21.057061 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:21.057017    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-68e64745-mq7qm"]
Oct 30 14:03:21.881730 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:21.881687    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d9c1728-a4ae-424d-a941-6a3e4e4a406b" path="/var/lib/kubelet/pods/0d9c1728-a4ae-424d-a941-6a3e4e4a406b/volumes"
Oct 30 14:03:23.158366 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:23.158324    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"]
Oct 30 14:03:28.159119 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:28.159069    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" podUID="86c76eb8-94da-4b75-b304-399be0b1c313" containerName="payload-pre-processing" containerID="cri-o://76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab" gracePeriod=25
Oct 30 14:03:28.922429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:28.922390    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" podUID="86c76eb8-94da-4b75-b304-399be0b1c313" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:03:38.253904 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:03:38.253860    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:03:38.929128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:38.929085    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" podUID="86c76eb8-94da-4b75-b304-399be0b1c313" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:03:48.460565 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:48.460536    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 14:03:48.594406 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:48.594364    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86c76eb8-94da-4b75-b304-399be0b1c313-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c76eb8-94da-4b75-b304-399be0b1c313-plugins-config-volume\") pod \"86c76eb8-94da-4b75-b304-399be0b1c313\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") "
Oct 30 14:03:48.594591 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:48.594501    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86c76eb8-94da-4b75-b304-399be0b1c313-kube-api-access-zlr95\" (UniqueName: \"kubernetes.io/projected/86c76eb8-94da-4b75-b304-399be0b1c313-kube-api-access-zlr95\") pod \"86c76eb8-94da-4b75-b304-399be0b1c313\" (UID: \"86c76eb8-94da-4b75-b304-399be0b1c313\") "
Oct 30 14:03:48.594893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:48.594863    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86c76eb8-94da-4b75-b304-399be0b1c313-plugins-config-volume" pod "86c76eb8-94da-4b75-b304-399be0b1c313" (UID: "86c76eb8-94da-4b75-b304-399be0b1c313"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:03:48.595199 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:48.595181    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86c76eb8-94da-4b75-b304-399be0b1c313-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:03:48.596796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:48.596770    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86c76eb8-94da-4b75-b304-399be0b1c313-kube-api-access-zlr95" pod "86c76eb8-94da-4b75-b304-399be0b1c313" (UID: "86c76eb8-94da-4b75-b304-399be0b1c313"). InnerVolumeSpecName "kube-api-access-zlr95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:03:48.696364 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:48.696249    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zlr95\" (UniqueName: \"kubernetes.io/projected/86c76eb8-94da-4b75-b304-399be0b1c313-kube-api-access-zlr95\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:03:49.240821 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.240787    4291 generic.go:356] "Generic (PLEG): container finished" podID="86c76eb8-94da-4b75-b304-399be0b1c313" containerID="76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab" exitCode=0
Oct 30 14:03:49.241025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.240855    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"
Oct 30 14:03:49.241025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.240880    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" event={"ID":"86c76eb8-94da-4b75-b304-399be0b1c313","Type":"ContainerDied","Data":"76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab"}
Oct 30 14:03:49.241025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.240948    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl" event={"ID":"86c76eb8-94da-4b75-b304-399be0b1c313","Type":"ContainerDied","Data":"8c5a51cb4d9325b2bd48c9bf6a9e4d6674961523c380666d1e10bbe3556b8b33"}
Oct 30 14:03:49.241025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.240981    4291 scope.go:122] "RemoveContainer" containerID="76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab"
Oct 30 14:03:49.253549 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.253446    4291 scope.go:122] "RemoveContainer" containerID="76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab"
Oct 30 14:03:49.253892 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:03:49.253859    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab\": container with ID starting with 76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab not found: ID does not exist" containerID="76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab"
Oct 30 14:03:49.254018 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.253902    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab"} err="failed to get container status \"76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab\": rpc error: code = NotFound desc = could not find container \"76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab\": container with ID starting with 76ec6fd040a128e716850dd9c52847b324a71fbd1a62e7cc135d3074d82267ab not found: ID does not exist"
Oct 30 14:03:49.302861 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.302824    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"]
Oct 30 14:03:49.315456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.315412    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-d724c8-6ddbb79bd7-sgqvl"]
Oct 30 14:03:49.882913 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:49.882884    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86c76eb8-94da-4b75-b304-399be0b1c313" path="/var/lib/kubelet/pods/86c76eb8-94da-4b75-b304-399be0b1c313/volumes"
Oct 30 14:03:56.897377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.897280    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"]
Oct 30 14:03:56.898741 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.898712    4291 cpu_manager.go:414] "containerMap: removing container" podUID="86c76eb8-94da-4b75-b304-399be0b1c313" containerName="payload-pre-processing"
Oct 30 14:03:56.898741 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.898741    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86c76eb8-94da-4b75-b304-399be0b1c313" containerName="payload-pre-processing"
Oct 30 14:03:56.898965 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.898797    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" containerName="revoke-keys"
Oct 30 14:03:56.898965 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.898809    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" containerName="revoke-keys"
Oct 30 14:03:56.899151 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.899133    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="86c76eb8-94da-4b75-b304-399be0b1c313" containerName="payload-pre-processing"
Oct 30 14:03:56.899244 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.899167    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" containerName="revoke-keys"
Oct 30 14:03:56.905740 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.905709    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:56.927994 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.927964    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"]
Oct 30 14:03:56.935718 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.935679    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:03:56.935919 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.935808    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:03:56.935919 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.935829    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:03:56.936150 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:56.936131    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:03:57.017735 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.017697    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.017944 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.017846    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ldn9\" (UniqueName: \"kubernetes.io/projected/72f18455-dcbe-4ada-a264-e01bc1d10a2d-kube-api-access-5ldn9\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.017944 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.017922    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.119257 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.119223    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.119478 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.119308    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ldn9\" (UniqueName: \"kubernetes.io/projected/72f18455-dcbe-4ada-a264-e01bc1d10a2d-kube-api-access-5ldn9\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.119478 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.119363    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.120035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.120002    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.121984 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.121943    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.145539 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.145506    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ldn9\" (UniqueName: \"kubernetes.io/projected/72f18455-dcbe-4ada-a264-e01bc1d10a2d-kube-api-access-5ldn9\") pod \"maas-api-revoke-keys-e2e-extmod-a-310546-plmxh\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.220492 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.220405    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:03:57.541500 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:57.541463    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"]
Oct 30 14:03:57.542258 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:03:57.542226    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f18455_dcbe_4ada_a264_e01bc1d10a2d.slice/crio-aa0de6d306ccf9ff41da18aae7d3fa994d846452bd827c862f4c0fe9109247cb WatchSource:0}: Error finding container aa0de6d306ccf9ff41da18aae7d3fa994d846452bd827c862f4c0fe9109247cb: Status 404 returned error can't find the container with id aa0de6d306ccf9ff41da18aae7d3fa994d846452bd827c862f4c0fe9109247cb
Oct 30 14:03:58.347648 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:58.347592    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh" event={"ID":"72f18455-dcbe-4ada-a264-e01bc1d10a2d","Type":"ContainerStarted","Data":"cce8e33f6f2a774a7e4c970a7a41a7660943c557881c6a44a5a89d416cfcc038"}
Oct 30 14:03:58.348344 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:58.348294    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh" event={"ID":"72f18455-dcbe-4ada-a264-e01bc1d10a2d","Type":"ContainerStarted","Data":"aa0de6d306ccf9ff41da18aae7d3fa994d846452bd827c862f4c0fe9109247cb"}
Oct 30 14:03:58.392506 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:58.392445    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh" podStartSLOduration=2.392428431 podStartE2EDuration="2.392428431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:03:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:03:58.389371293 +0000 UTC m=+3648.649609528" watchObservedRunningTime="2026-10-30 14:03:58.392428431 +0000 UTC m=+3648.652666715"
Oct 30 14:03:59.357499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:59.357466    4291 generic.go:356] "Generic (PLEG): container finished" podID="72f18455-dcbe-4ada-a264-e01bc1d10a2d" containerID="cce8e33f6f2a774a7e4c970a7a41a7660943c557881c6a44a5a89d416cfcc038" exitCode=0
Oct 30 14:03:59.357945 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:03:59.357562    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh" event={"ID":"72f18455-dcbe-4ada-a264-e01bc1d10a2d","Type":"ContainerDied","Data":"cce8e33f6f2a774a7e4c970a7a41a7660943c557881c6a44a5a89d416cfcc038"}
Oct 30 14:04:00.539774 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.539745    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:04:00.681319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.681227    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f18455-dcbe-4ada-a264-e01bc1d10a2d-kube-api-access-5ldn9\" (UniqueName: \"kubernetes.io/projected/72f18455-dcbe-4ada-a264-e01bc1d10a2d-kube-api-access-5ldn9\") pod \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") "
Oct 30 14:04:00.681507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.681356    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-openshift-ca\") pod \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") "
Oct 30 14:04:00.681507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.681437    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-xks-ca\") pod \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\" (UID: \"72f18455-dcbe-4ada-a264-e01bc1d10a2d\") "
Oct 30 14:04:00.681771 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.681743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-openshift-ca" pod "72f18455-dcbe-4ada-a264-e01bc1d10a2d" (UID: "72f18455-dcbe-4ada-a264-e01bc1d10a2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:00.682056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.682033    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:00.683318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.683297    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f18455-dcbe-4ada-a264-e01bc1d10a2d-kube-api-access-5ldn9" pod "72f18455-dcbe-4ada-a264-e01bc1d10a2d" (UID: "72f18455-dcbe-4ada-a264-e01bc1d10a2d"). InnerVolumeSpecName "kube-api-access-5ldn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:00.683460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.683444    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-xks-ca" pod "72f18455-dcbe-4ada-a264-e01bc1d10a2d" (UID: "72f18455-dcbe-4ada-a264-e01bc1d10a2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:00.783292 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.783256    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f18455-dcbe-4ada-a264-e01bc1d10a2d-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:00.783292 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:00.783284    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5ldn9\" (UniqueName: \"kubernetes.io/projected/72f18455-dcbe-4ada-a264-e01bc1d10a2d-kube-api-access-5ldn9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:01.373571 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:01.373531    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh" event={"ID":"72f18455-dcbe-4ada-a264-e01bc1d10a2d","Type":"ContainerDied","Data":"aa0de6d306ccf9ff41da18aae7d3fa994d846452bd827c862f4c0fe9109247cb"}
Oct 30 14:04:01.373773 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:01.373580    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa0de6d306ccf9ff41da18aae7d3fa994d846452bd827c862f4c0fe9109247cb"
Oct 30 14:04:01.373773 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:01.373549    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"
Oct 30 14:04:07.819406 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:07.819340    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:04:08.212707 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:04:08.212667    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:04:09.468213 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:09.468172    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"]
Oct 30 14:04:09.479754 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:09.479717    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-d724c8-zjccc"]
Oct 30 14:04:09.881337 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:09.881298    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4" path="/var/lib/kubelet/pods/ef0dd223-2bbf-4d34-ac86-38c7a9e16bd4/volumes"
Oct 30 14:04:14.103365 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:14.103333    4291 scope.go:122] "RemoveContainer" containerID="5bd46f446dba6a6cfc68602d43997ca9c556b58011ce2801ca9747ccd770e2b1"
Oct 30 14:04:14.115256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:14.115234    4291 scope.go:122] "RemoveContainer" containerID="77e85a0ac8e852eff1b9303a3ba4a60e4c4102f7fa01f7eece18ffaf5a8233a4"
Oct 30 14:04:14.129832 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:14.129805    4291 scope.go:122] "RemoveContainer" containerID="906da1402bc7fbecee17d5654705cb575fb9c1c1d11369c789c785e250adf7ab"
Oct 30 14:04:19.630902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.630866    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"]
Oct 30 14:04:19.631489 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.631473    4291 cpu_manager.go:414] "containerMap: removing container" podUID="72f18455-dcbe-4ada-a264-e01bc1d10a2d" containerName="revoke-keys"
Oct 30 14:04:19.631536 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.631490    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f18455-dcbe-4ada-a264-e01bc1d10a2d" containerName="revoke-keys"
Oct 30 14:04:19.631625 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.631616    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f18455-dcbe-4ada-a264-e01bc1d10a2d" containerName="revoke-keys"
Oct 30 14:04:19.635341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.635319    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.654669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.650767    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:04:19.654669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.650868    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:04:19.655121 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.655094    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"]
Oct 30 14:04:19.664679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.662972    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:04:19.664679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.662984    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:04:19.714794 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.714759    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.714935 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.714808    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.714935 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.714914    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq6h8\" (UniqueName: \"kubernetes.io/projected/89fa2b6d-96da-4981-b628-ff436c2d0f05-kube-api-access-lq6h8\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.815849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.815803    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.816038 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.815856    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.816038 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.815889    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq6h8\" (UniqueName: \"kubernetes.io/projected/89fa2b6d-96da-4981-b628-ff436c2d0f05-kube-api-access-lq6h8\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.816664 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.816608    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.818406 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.818383    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.837232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.837206    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq6h8\" (UniqueName: \"kubernetes.io/projected/89fa2b6d-96da-4981-b628-ff436c2d0f05-kube-api-access-lq6h8\") pod \"maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:19.951857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:19.951773    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:20.270015 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:04:20.269969    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89fa2b6d_96da_4981_b628_ff436c2d0f05.slice/crio-f50628abaf62bc93be7d0c21caf58e210a1b04a58f6524d066978bd6e16d7de3 WatchSource:0}: Error finding container f50628abaf62bc93be7d0c21caf58e210a1b04a58f6524d066978bd6e16d7de3: Status 404 returned error can't find the container with id f50628abaf62bc93be7d0c21caf58e210a1b04a58f6524d066978bd6e16d7de3
Oct 30 14:04:20.270252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:20.270230    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"]
Oct 30 14:04:20.530953 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:20.530874    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp" event={"ID":"89fa2b6d-96da-4981-b628-ff436c2d0f05","Type":"ContainerStarted","Data":"f50628abaf62bc93be7d0c21caf58e210a1b04a58f6524d066978bd6e16d7de3"}
Oct 30 14:04:21.539569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:21.539473    4291 generic.go:356] "Generic (PLEG): container finished" podID="89fa2b6d-96da-4981-b628-ff436c2d0f05" containerID="fa86c13b397e233d1a386b7bfc7e16765e41c652db5cfca039f0594137e4aa4c" exitCode=0
Oct 30 14:04:21.539569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:21.539511    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp" event={"ID":"89fa2b6d-96da-4981-b628-ff436c2d0f05","Type":"ContainerDied","Data":"fa86c13b397e233d1a386b7bfc7e16765e41c652db5cfca039f0594137e4aa4c"}
Oct 30 14:04:22.702414 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.702388    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:22.765851 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.765811    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-xks-ca\") pod \"89fa2b6d-96da-4981-b628-ff436c2d0f05\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") "
Oct 30 14:04:22.783033 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.776963    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-openshift-ca\") pod \"89fa2b6d-96da-4981-b628-ff436c2d0f05\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") "
Oct 30 14:04:22.783481 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.783437    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89fa2b6d-96da-4981-b628-ff436c2d0f05-kube-api-access-lq6h8\" (UniqueName: \"kubernetes.io/projected/89fa2b6d-96da-4981-b628-ff436c2d0f05-kube-api-access-lq6h8\") pod \"89fa2b6d-96da-4981-b628-ff436c2d0f05\" (UID: \"89fa2b6d-96da-4981-b628-ff436c2d0f05\") "
Oct 30 14:04:22.785145 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.780192    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-openshift-ca" pod "89fa2b6d-96da-4981-b628-ff436c2d0f05" (UID: "89fa2b6d-96da-4981-b628-ff436c2d0f05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:22.799720 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.799648    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-xks-ca" pod "89fa2b6d-96da-4981-b628-ff436c2d0f05" (UID: "89fa2b6d-96da-4981-b628-ff436c2d0f05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:22.816793 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.811456    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89fa2b6d-96da-4981-b628-ff436c2d0f05-kube-api-access-lq6h8" pod "89fa2b6d-96da-4981-b628-ff436c2d0f05" (UID: "89fa2b6d-96da-4981-b628-ff436c2d0f05"). InnerVolumeSpecName "kube-api-access-lq6h8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:22.887125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.887086    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:22.887125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.887125    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/89fa2b6d-96da-4981-b628-ff436c2d0f05-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:22.887125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:22.887138    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq6h8\" (UniqueName: \"kubernetes.io/projected/89fa2b6d-96da-4981-b628-ff436c2d0f05-kube-api-access-lq6h8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:23.555765 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:23.555727    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp" event={"ID":"89fa2b6d-96da-4981-b628-ff436c2d0f05","Type":"ContainerDied","Data":"f50628abaf62bc93be7d0c21caf58e210a1b04a58f6524d066978bd6e16d7de3"}
Oct 30 14:04:23.555765 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:23.555753    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"
Oct 30 14:04:23.556006 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:23.555780    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f50628abaf62bc93be7d0c21caf58e210a1b04a58f6524d066978bd6e16d7de3"
Oct 30 14:04:30.020723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:30.020684    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"]
Oct 30 14:04:30.050774 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:30.050734    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"]
Oct 30 14:04:33.843515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.843472    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"]
Oct 30 14:04:33.844414 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.844387    4291 cpu_manager.go:414] "containerMap: removing container" podUID="89fa2b6d-96da-4981-b628-ff436c2d0f05" containerName="revoke-keys"
Oct 30 14:04:33.844414 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.844415    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89fa2b6d-96da-4981-b628-ff436c2d0f05" containerName="revoke-keys"
Oct 30 14:04:33.844587 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.844574    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="89fa2b6d-96da-4981-b628-ff436c2d0f05" containerName="revoke-keys"
Oct 30 14:04:33.848867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.848841    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.881232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.881193    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:04:33.881452 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.881322    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:04:33.881452 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.881374    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:04:33.881452 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.881439    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:04:33.882221 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.882200    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"]
Oct 30 14:04:33.888453 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.888397    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gl9m\" (UniqueName: \"kubernetes.io/projected/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-kube-api-access-5gl9m\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.888715 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.888464    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.888860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.888837    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.990333 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.990280    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.990611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.990591    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.990739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.990696    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gl9m\" (UniqueName: \"kubernetes.io/projected/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-kube-api-access-5gl9m\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.991377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.991351    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:33.993500 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:33.993475    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:34.012100 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:34.012070    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gl9m\" (UniqueName: \"kubernetes.io/projected/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-kube-api-access-5gl9m\") pod \"maas-api-revoke-sub-73641705-lzdjc\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:34.164554 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:34.164457    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:34.482057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:34.482015    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"]
Oct 30 14:04:34.483332 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:04:34.483301    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f9c7fac_d05f_4f32_bc4f_e1fe45c115f0.slice/crio-7ac8a2f0cfe2d4865e10d5f990a716d5005a88243392cb278e9018b9d6b5ada0 WatchSource:0}: Error finding container 7ac8a2f0cfe2d4865e10d5f990a716d5005a88243392cb278e9018b9d6b5ada0: Status 404 returned error can't find the container with id 7ac8a2f0cfe2d4865e10d5f990a716d5005a88243392cb278e9018b9d6b5ada0
Oct 30 14:04:34.692601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:34.692557    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc" event={"ID":"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0","Type":"ContainerStarted","Data":"7ac8a2f0cfe2d4865e10d5f990a716d5005a88243392cb278e9018b9d6b5ada0"}
Oct 30 14:04:35.058078 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.052728    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" podUID="47798c3a-363a-4a27-9148-ed8caf915e1a" containerName="payload-pre-processing" containerID="cri-o://8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a" gracePeriod=25
Oct 30 14:04:35.618812 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.618769    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"]
Oct 30 14:04:35.631409 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.631357    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-310546-plmxh"]
Oct 30 14:04:35.661664 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.652710    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd" podUID="e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" containerName="payload-processing" containerID="cri-o://32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9" gracePeriod=25
Oct 30 14:04:35.705505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.705464    4291 generic.go:356] "Generic (PLEG): container finished" podID="9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" containerID="b26d1a5b9856d76ef26c97433f3bb4c716b97e7e88f0adf9bf78713e86cdc93e" exitCode=0
Oct 30 14:04:35.705681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.705537    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc" event={"ID":"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0","Type":"ContainerDied","Data":"b26d1a5b9856d76ef26c97433f3bb4c716b97e7e88f0adf9bf78713e86cdc93e"}
Oct 30 14:04:35.890217 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.890180    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f18455-dcbe-4ada-a264-e01bc1d10a2d" path="/var/lib/kubelet/pods/72f18455-dcbe-4ada-a264-e01bc1d10a2d/volumes"
Oct 30 14:04:35.970133 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:35.970099    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 14:04:36.068271 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.068121    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-kube-api-access-8ffw5\" (UniqueName: \"kubernetes.io/projected/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-kube-api-access-8ffw5\") pod \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") "
Oct 30 14:04:36.068778 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.068313    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-plugins-config-volume\") pod \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\" (UID: \"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d\") "
Oct 30 14:04:36.069096 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.069002    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-plugins-config-volume" pod "e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" (UID: "e3e9026a-cb2f-45b9-ae82-aaf7dad1075d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:36.071217 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.071104    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-kube-api-access-8ffw5" pod "e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" (UID: "e3e9026a-cb2f-45b9-ae82-aaf7dad1075d"). InnerVolumeSpecName "kube-api-access-8ffw5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:36.170428 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.170387    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ffw5\" (UniqueName: \"kubernetes.io/projected/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-kube-api-access-8ffw5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:36.170428 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.170424    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:36.715107 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.715071    4291 generic.go:356] "Generic (PLEG): container finished" podID="e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" containerID="32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9" exitCode=0
Oct 30 14:04:36.715323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.715119    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"
Oct 30 14:04:36.715323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.715150    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd" event={"ID":"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d","Type":"ContainerDied","Data":"32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9"}
Oct 30 14:04:36.715323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.715207    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd" event={"ID":"e3e9026a-cb2f-45b9-ae82-aaf7dad1075d","Type":"ContainerDied","Data":"f55ceb8d38f2a06b255d53551e80b16566da8a1235eab364da15dbb77a856260"}
Oct 30 14:04:36.715323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.715232    4291 scope.go:122] "RemoveContainer" containerID="32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9"
Oct 30 14:04:36.728685 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.728662    4291 scope.go:122] "RemoveContainer" containerID="32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9"
Oct 30 14:04:36.729172 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:04:36.729146    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9\": container with ID starting with 32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9 not found: ID does not exist" containerID="32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9"
Oct 30 14:04:36.729246 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.729186    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9"} err="failed to get container status \"32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9\": rpc error: code = NotFound desc = could not find container \"32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9\": container with ID starting with 32986657a13df0a6eb87967823f30f4070e596eec8e7100265278bd1419c3dc9 not found: ID does not exist"
Oct 30 14:04:36.766971 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.766933    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"]
Oct 30 14:04:36.780876 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.780844    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-ca0243-849ccc47b-pnhzd"]
Oct 30 14:04:36.898755 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:36.898731    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:37.084836 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.084743    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-openshift-ca\") pod \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") "
Oct 30 14:04:37.085329 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.084879    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-kube-api-access-5gl9m\" (UniqueName: \"kubernetes.io/projected/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-kube-api-access-5gl9m\") pod \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") "
Oct 30 14:04:37.085329 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.084989    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-xks-ca\") pod \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\" (UID: \"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0\") "
Oct 30 14:04:37.085329 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.085064    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-openshift-ca" pod "9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" (UID: "9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:37.085524 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.085486    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:37.087123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.087099    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-xks-ca" pod "9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" (UID: "9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:37.087250 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.087183    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-kube-api-access-5gl9m" pod "9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" (UID: "9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0"). InnerVolumeSpecName "kube-api-access-5gl9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:37.186725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.186681    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:37.186725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.186717    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gl9m\" (UniqueName: \"kubernetes.io/projected/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0-kube-api-access-5gl9m\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:37.727722 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.727683    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"
Oct 30 14:04:37.727722 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.727700    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc" event={"ID":"9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0","Type":"ContainerDied","Data":"7ac8a2f0cfe2d4865e10d5f990a716d5005a88243392cb278e9018b9d6b5ada0"}
Oct 30 14:04:37.727998 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.727760    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ac8a2f0cfe2d4865e10d5f990a716d5005a88243392cb278e9018b9d6b5ada0"
Oct 30 14:04:37.926975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:37.926936    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" path="/var/lib/kubelet/pods/e3e9026a-cb2f-45b9-ae82-aaf7dad1075d/volumes"
Oct 30 14:04:38.211559 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:04:38.211519    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:04:40.035616 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:40.035584    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"]
Oct 30 14:04:40.047176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:40.047141    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f18ed421-zqx78"]
Oct 30 14:04:40.282132 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:40.282093    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" podUID="47798c3a-363a-4a27-9148-ed8caf915e1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:04:41.881516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:41.881467    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92b697ce-2854-42ce-95c2-cd1993de2adf" path="/var/lib/kubelet/pods/92b697ce-2854-42ce-95c2-cd1993de2adf/volumes"
Oct 30 14:04:44.172940 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.172903    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"]
Oct 30 14:04:44.173652 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.173620    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" containerName="payload-processing"
Oct 30 14:04:44.173723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.173655    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" containerName="payload-processing"
Oct 30 14:04:44.173723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.173687    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" containerName="revoke-keys"
Oct 30 14:04:44.173723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.173692    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" containerName="revoke-keys"
Oct 30 14:04:44.173820 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.173801    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e3e9026a-cb2f-45b9-ae82-aaf7dad1075d" containerName="payload-processing"
Oct 30 14:04:44.173820 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.173814    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" containerName="revoke-keys"
Oct 30 14:04:44.178522 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.178502    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.195537 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.195510    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"]
Oct 30 14:04:44.203206 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.203183    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:04:44.203206 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.203196    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:04:44.236449 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.236418    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hcfn\" (UniqueName: \"kubernetes.io/projected/3e5ac6a7-7352-42e4-8214-13f674589a0c-kube-api-access-8hcfn\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.236608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.236475    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.236704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.236684    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.286401 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.286362    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:04:44.286556 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.286413    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:04:44.338185 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.338152    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hcfn\" (UniqueName: \"kubernetes.io/projected/3e5ac6a7-7352-42e4-8214-13f674589a0c-kube-api-access-8hcfn\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.338352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.338204    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.338352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.338312    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.338974 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.338954    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.340747 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.340728    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.358795 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.358771    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hcfn\" (UniqueName: \"kubernetes.io/projected/3e5ac6a7-7352-42e4-8214-13f674589a0c-kube-api-access-8hcfn\") pod \"maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.492540 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.492461    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:44.797908 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.797882    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"]
Oct 30 14:04:44.799938 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:04:44.799893    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e5ac6a7_7352_42e4_8214_13f674589a0c.slice/crio-54d3c5d73dd82ce7542449bc1a01a3bb0464b7d7b044a9e7e4409ac9e53b9719 WatchSource:0}: Error finding container 54d3c5d73dd82ce7542449bc1a01a3bb0464b7d7b044a9e7e4409ac9e53b9719: Status 404 returned error can't find the container with id 54d3c5d73dd82ce7542449bc1a01a3bb0464b7d7b044a9e7e4409ac9e53b9719
Oct 30 14:04:44.826703 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:44.826676    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x" event={"ID":"3e5ac6a7-7352-42e4-8214-13f674589a0c","Type":"ContainerStarted","Data":"54d3c5d73dd82ce7542449bc1a01a3bb0464b7d7b044a9e7e4409ac9e53b9719"}
Oct 30 14:04:45.836189 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:45.836155    4291 generic.go:356] "Generic (PLEG): container finished" podID="3e5ac6a7-7352-42e4-8214-13f674589a0c" containerID="337af867a438c6a0598ec76c2814def90da059dffcec725c5092240fab533d0c" exitCode=0
Oct 30 14:04:45.836612 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:45.836248    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x" event={"ID":"3e5ac6a7-7352-42e4-8214-13f674589a0c","Type":"ContainerDied","Data":"337af867a438c6a0598ec76c2814def90da059dffcec725c5092240fab533d0c"}
Oct 30 14:04:47.028844 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.028817    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:47.175990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.175904    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-openshift-ca\") pod \"3e5ac6a7-7352-42e4-8214-13f674589a0c\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") "
Oct 30 14:04:47.176148 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.176096    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e5ac6a7-7352-42e4-8214-13f674589a0c-kube-api-access-8hcfn\" (UniqueName: \"kubernetes.io/projected/3e5ac6a7-7352-42e4-8214-13f674589a0c-kube-api-access-8hcfn\") pod \"3e5ac6a7-7352-42e4-8214-13f674589a0c\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") "
Oct 30 14:04:47.176331 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.176309    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-openshift-ca" pod "3e5ac6a7-7352-42e4-8214-13f674589a0c" (UID: "3e5ac6a7-7352-42e4-8214-13f674589a0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:47.176371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.176324    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-xks-ca\") pod \"3e5ac6a7-7352-42e4-8214-13f674589a0c\" (UID: \"3e5ac6a7-7352-42e4-8214-13f674589a0c\") "
Oct 30 14:04:47.176824 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.176803    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:47.178311 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.178281    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e5ac6a7-7352-42e4-8214-13f674589a0c-kube-api-access-8hcfn" pod "3e5ac6a7-7352-42e4-8214-13f674589a0c" (UID: "3e5ac6a7-7352-42e4-8214-13f674589a0c"). InnerVolumeSpecName "kube-api-access-8hcfn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:47.178425 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.178317    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-xks-ca" pod "3e5ac6a7-7352-42e4-8214-13f674589a0c" (UID: "3e5ac6a7-7352-42e4-8214-13f674589a0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:04:47.277709 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.277631    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3e5ac6a7-7352-42e4-8214-13f674589a0c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:47.277709 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.277703    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hcfn\" (UniqueName: \"kubernetes.io/projected/3e5ac6a7-7352-42e4-8214-13f674589a0c-kube-api-access-8hcfn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:47.854051 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.854017    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"
Oct 30 14:04:47.854051 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.854045    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x" event={"ID":"3e5ac6a7-7352-42e4-8214-13f674589a0c","Type":"ContainerDied","Data":"54d3c5d73dd82ce7542449bc1a01a3bb0464b7d7b044a9e7e4409ac9e53b9719"}
Oct 30 14:04:47.854267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:47.854095    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="54d3c5d73dd82ce7542449bc1a01a3bb0464b7d7b044a9e7e4409ac9e53b9719"
Oct 30 14:04:50.281624 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:50.281578    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" podUID="47798c3a-363a-4a27-9148-ed8caf915e1a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:04:52.032692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:52.032649    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"]
Oct 30 14:04:52.044226 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:52.044190    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1019287e-82227"]
Oct 30 14:04:53.879585 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:53.879551    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b8479a-41f2-40ad-9cbc-9300dc7bcd9c" path="/var/lib/kubelet/pods/88b8479a-41f2-40ad-9cbc-9300dc7bcd9c/volumes"
Oct 30 14:04:54.569424 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:54.569387    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"]
Oct 30 14:04:55.438402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.438375    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 14:04:55.546323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.546292    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47798c3a-363a-4a27-9148-ed8caf915e1a-kube-api-access-4xmbx\" (UniqueName: \"kubernetes.io/projected/47798c3a-363a-4a27-9148-ed8caf915e1a-kube-api-access-4xmbx\") pod \"47798c3a-363a-4a27-9148-ed8caf915e1a\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") "
Oct 30 14:04:55.546504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.546470    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47798c3a-363a-4a27-9148-ed8caf915e1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47798c3a-363a-4a27-9148-ed8caf915e1a-plugins-config-volume\") pod \"47798c3a-363a-4a27-9148-ed8caf915e1a\" (UID: \"47798c3a-363a-4a27-9148-ed8caf915e1a\") "
Oct 30 14:04:55.546867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.546841    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47798c3a-363a-4a27-9148-ed8caf915e1a-plugins-config-volume" pod "47798c3a-363a-4a27-9148-ed8caf915e1a" (UID: "47798c3a-363a-4a27-9148-ed8caf915e1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:04:55.547055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.547041    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47798c3a-363a-4a27-9148-ed8caf915e1a-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:55.548315 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.548295    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47798c3a-363a-4a27-9148-ed8caf915e1a-kube-api-access-4xmbx" pod "47798c3a-363a-4a27-9148-ed8caf915e1a" (UID: "47798c3a-363a-4a27-9148-ed8caf915e1a"). InnerVolumeSpecName "kube-api-access-4xmbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:04:55.648696 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.648657    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmbx\" (UniqueName: \"kubernetes.io/projected/47798c3a-363a-4a27-9148-ed8caf915e1a-kube-api-access-4xmbx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:04:55.916569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.916481    4291 generic.go:356] "Generic (PLEG): container finished" podID="47798c3a-363a-4a27-9148-ed8caf915e1a" containerID="8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a" exitCode=0
Oct 30 14:04:55.916569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.916528    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"
Oct 30 14:04:55.916569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.916560    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" event={"ID":"47798c3a-363a-4a27-9148-ed8caf915e1a","Type":"ContainerDied","Data":"8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a"}
Oct 30 14:04:55.916881 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.916617    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v" event={"ID":"47798c3a-363a-4a27-9148-ed8caf915e1a","Type":"ContainerDied","Data":"4356bb04d6b6beefe9af2ef62c6cbfcdf76faeebdec4969aed65a0852149fc1d"}
Oct 30 14:04:55.916881 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.916661    4291 scope.go:122] "RemoveContainer" containerID="8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a"
Oct 30 14:04:55.928678 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.928661    4291 scope.go:122] "RemoveContainer" containerID="8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a"
Oct 30 14:04:55.928947 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:04:55.928928    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a\": container with ID starting with 8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a not found: ID does not exist" containerID="8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a"
Oct 30 14:04:55.929024 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.928961    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a"} err="failed to get container status \"8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a\": rpc error: code = NotFound desc = could not find container \"8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a\": container with ID starting with 8f414344f95dc4b32d7864c8adadf497cb0624fb5f2689c80ad0ca6d5ac6553a not found: ID does not exist"
Oct 30 14:04:55.968368 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.968338    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"]
Oct 30 14:04:55.979934 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:55.979906    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-ca0243-7cfc4c7b79-zkk8v"]
Oct 30 14:04:57.880820 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:57.880779    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47798c3a-363a-4a27-9148-ed8caf915e1a" path="/var/lib/kubelet/pods/47798c3a-363a-4a27-9148-ed8caf915e1a/volumes"
Oct 30 14:04:59.570752 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:04:59.570702    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" podUID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" containerName="payload-processing" containerID="cri-o://df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02" gracePeriod=25
Oct 30 14:05:03.738093 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:03.738052    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" podUID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:05:06.024321 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:06.024284    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"]
Oct 30 14:05:06.035221 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:06.035193    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-ca0243-2gxhp"]
Oct 30 14:05:07.919668 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:07.919014    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89fa2b6d-96da-4981-b628-ff436c2d0f05" path="/var/lib/kubelet/pods/89fa2b6d-96da-4981-b628-ff436c2d0f05/volumes"
Oct 30 14:05:07.919668 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:07.919594    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:05:08.210015 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:05:08.209925    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:05:13.737780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:13.737740    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" podUID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:05:14.225233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:14.225197    4291 scope.go:122] "RemoveContainer" containerID="1e3453ca8a88d188e46c027917e73bb11a68d434ee8f51819d50df2ea71368f6"
Oct 30 14:05:14.240129 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:14.237410    4291 scope.go:122] "RemoveContainer" containerID="37d0d3313d51ecffb5c9b0f270417605c689b7a91704563f69f58c13f7f9bf9c"
Oct 30 14:05:14.250052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:14.250032    4291 scope.go:122] "RemoveContainer" containerID="9eb6a5ad4e8d33fcb0fabef8980bf681214e75ed3b9161182f1cda56b44368c1"
Oct 30 14:05:14.655735 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:14.655701    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"]
Oct 30 14:05:14.656044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:14.656002    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy" containerID="cri-o://dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0" gracePeriod=30
Oct 30 14:05:19.951502 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:19.951476    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 14:05:19.976326 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:19.976303    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 14:05:20.117123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117039    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-envoy\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117083    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-certs\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117135    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-credential-socket\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117222    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-istio-token\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-istio-token\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117285    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c272e255-8e03-4edb-8dd2-50b0fb50c19d-kube-api-access-qg7m4\" (UniqueName: \"kubernetes.io/projected/c272e255-8e03-4edb-8dd2-50b0fb50c19d-kube-api-access-qg7m4\") pod \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") "
Oct 30 14:05:20.117371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117353    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-socket\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117599 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117428    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-certs" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:20.117599 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117436    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c272e255-8e03-4edb-8dd2-50b0fb50c19d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c272e255-8e03-4edb-8dd2-50b0fb50c19d-plugins-config-volume\") pod \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\" (UID: \"c272e255-8e03-4edb-8dd2-50b0fb50c19d\") "
Oct 30 14:05:20.117599 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117526    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-credential-socket" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:20.117599 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117568    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-data\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117686    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bafd4569-5128-4522-86f8-12b44c954428-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd4569-5128-4522-86f8-12b44c954428-istiod-ca-cert\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-socket" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:20.117849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117762    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-kube-api-access-l5wc4\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-kube-api-access-l5wc4\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.117849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117805    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bafd4569-5128-4522-86f8-12b44c954428-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd4569-5128-4522-86f8-12b44c954428-istio-podinfo\") pod \"bafd4569-5128-4522-86f8-12b44c954428\" (UID: \"bafd4569-5128-4522-86f8-12b44c954428\") "
Oct 30 14:05:20.118066 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.117863    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-data" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:20.118122 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.118070    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bafd4569-5128-4522-86f8-12b44c954428-istiod-ca-cert" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:05:20.118482 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.118452    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c272e255-8e03-4edb-8dd2-50b0fb50c19d-plugins-config-volume" pod "c272e255-8e03-4edb-8dd2-50b0fb50c19d" (UID: "c272e255-8e03-4edb-8dd2-50b0fb50c19d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:05:20.118710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.118546    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.118864 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.118722    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bafd4569-5128-4522-86f8-12b44c954428-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.118864 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.118739    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.118864 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.118754    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.118864 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.118768    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.119711 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.119687    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-envoy" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:20.120255 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.120236    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-istio-token" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:20.120365 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.120276    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c272e255-8e03-4edb-8dd2-50b0fb50c19d-kube-api-access-qg7m4" pod "c272e255-8e03-4edb-8dd2-50b0fb50c19d" (UID: "c272e255-8e03-4edb-8dd2-50b0fb50c19d"). InnerVolumeSpecName "kube-api-access-qg7m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:20.120444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.120402    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-kube-api-access-l5wc4" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "kube-api-access-l5wc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:20.120978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.120956    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bafd4569-5128-4522-86f8-12b44c954428-istio-podinfo" pod "bafd4569-5128-4522-86f8-12b44c954428" (UID: "bafd4569-5128-4522-86f8-12b44c954428"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:05:20.172114 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.172090    4291 generic.go:356] "Generic (PLEG): container finished" podID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" containerID="df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02" exitCode=0
Oct 30 14:05:20.172237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.172136    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"
Oct 30 14:05:20.172237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.172179    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" event={"ID":"c272e255-8e03-4edb-8dd2-50b0fb50c19d","Type":"ContainerDied","Data":"df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02"}
Oct 30 14:05:20.172237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.172234    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p" event={"ID":"c272e255-8e03-4edb-8dd2-50b0fb50c19d","Type":"ContainerDied","Data":"759e29307d9713281e28ee0d028e0f9584bca5e1d555c42ceedf79e756799cba"}
Oct 30 14:05:20.172415 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.172262    4291 scope.go:122] "RemoveContainer" containerID="df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02"
Oct 30 14:05:20.174120 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.174093    4291 generic.go:356] "Generic (PLEG): container finished" podID="bafd4569-5128-4522-86f8-12b44c954428" containerID="dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0" exitCode=0
Oct 30 14:05:20.174242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.174144    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"
Oct 30 14:05:20.174242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.174177    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" event={"ID":"bafd4569-5128-4522-86f8-12b44c954428","Type":"ContainerDied","Data":"dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0"}
Oct 30 14:05:20.174242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.174231    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6" event={"ID":"bafd4569-5128-4522-86f8-12b44c954428","Type":"ContainerDied","Data":"1ff8aa9be2f60002b4481e2eea949f163f5ab3a85eb4640513d7c200cbc63c06"}
Oct 30 14:05:20.183566 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.183546    4291 scope.go:122] "RemoveContainer" containerID="df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02"
Oct 30 14:05:20.183878 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:05:20.183858    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02\": container with ID starting with df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02 not found: ID does not exist" containerID="df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02"
Oct 30 14:05:20.183946 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.183883    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02"} err="failed to get container status \"df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02\": rpc error: code = NotFound desc = could not find container \"df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02\": container with ID starting with df8e251054d8bbfc0ec8fe40dbc23b531d1fda9d36af70a705f53dfa08927a02 not found: ID does not exist"
Oct 30 14:05:20.183946 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.183900    4291 scope.go:122] "RemoveContainer" containerID="dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0"
Oct 30 14:05:20.193084 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.193066    4291 scope.go:122] "RemoveContainer" containerID="7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1"
Oct 30 14:05:20.202094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.202073    4291 scope.go:122] "RemoveContainer" containerID="dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0"
Oct 30 14:05:20.202333 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:05:20.202314    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0\": container with ID starting with dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0 not found: ID does not exist" containerID="dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0"
Oct 30 14:05:20.202417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.202342    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0"} err="failed to get container status \"dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0\": rpc error: code = NotFound desc = could not find container \"dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0\": container with ID starting with dc1de5b7f2e444d631ab92090f1d68e56b5311ebda526ef4cc8807fa69c930d0 not found: ID does not exist"
Oct 30 14:05:20.202417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.202364    4291 scope.go:122] "RemoveContainer" containerID="7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1"
Oct 30 14:05:20.202610 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:05:20.202592    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1\": container with ID starting with 7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1 not found: ID does not exist" containerID="7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1"
Oct 30 14:05:20.202674 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.202617    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1"} err="failed to get container status \"7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1\": rpc error: code = NotFound desc = could not find container \"7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1\": container with ID starting with 7f438aff76f4fbf0c90ebf00f77c99f1c9bcae751319550ecfddc6e5bf3bf7c1 not found: ID does not exist"
Oct 30 14:05:20.219917 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.219895    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5wc4\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-kube-api-access-l5wc4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.219917 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.219918    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bafd4569-5128-4522-86f8-12b44c954428-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.220064 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.219928    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bafd4569-5128-4522-86f8-12b44c954428-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.220064 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.219936    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bafd4569-5128-4522-86f8-12b44c954428-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.220064 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.219944    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg7m4\" (UniqueName: \"kubernetes.io/projected/c272e255-8e03-4edb-8dd2-50b0fb50c19d-kube-api-access-qg7m4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.220064 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.219952    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c272e255-8e03-4edb-8dd2-50b0fb50c19d-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:20.224233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.224213    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"]
Oct 30 14:05:20.227443 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.227424    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-e3eb98-7d5cd58446-cvf4p"]
Oct 30 14:05:20.259187 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.259165    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"]
Oct 30 14:05:20.265882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:20.265864    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-ca0243-openshift-default-966db98cf-4jqc6"]
Oct 30 14:05:21.880358 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:21.880323    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bafd4569-5128-4522-86f8-12b44c954428" path="/var/lib/kubelet/pods/bafd4569-5128-4522-86f8-12b44c954428/volumes"
Oct 30 14:05:21.880937 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:21.880919    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" path="/var/lib/kubelet/pods/c272e255-8e03-4edb-8dd2-50b0fb50c19d/volumes"
Oct 30 14:05:27.033144 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:27.033053    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"]
Oct 30 14:05:27.044484 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:27.044457    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-54c67a3e-qdbpg"]
Oct 30 14:05:27.881778 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:27.881743    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45f0dda2-ba36-42a4-bc8b-e67f329c007f" path="/var/lib/kubelet/pods/45f0dda2-ba36-42a4-bc8b-e67f329c007f/volumes"
Oct 30 14:05:34.210811 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:34.210766    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"]
Oct 30 14:05:34.222272 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:34.222242    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-e3eb98-nnt8x"]
Oct 30 14:05:35.880417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:35.880378    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e5ac6a7-7352-42e4-8214-13f674589a0c" path="/var/lib/kubelet/pods/3e5ac6a7-7352-42e4-8214-13f674589a0c/volumes"
Oct 30 14:05:38.031170 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:38.031126    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"]
Oct 30 14:05:38.042730 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:38.042702    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f095babd-p7t7n"]
Oct 30 14:05:38.211517 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:05:38.211480    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:05:39.892854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:39.892818    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e24cc45-8b37-4863-8189-8d0dc5b4eecf" path="/var/lib/kubelet/pods/5e24cc45-8b37-4863-8189-8d0dc5b4eecf/volumes"
Oct 30 14:05:41.474387 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:41.474350    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"]
Oct 30 14:05:41.474969 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:41.474649    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" containerID="cri-o://7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636" gracePeriod=30
Oct 30 14:05:42.062156 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062118    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"]
Oct 30 14:05:42.062725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062709    4291 cpu_manager.go:414] "containerMap: removing container" podUID="47798c3a-363a-4a27-9148-ed8caf915e1a" containerName="payload-pre-processing"
Oct 30 14:05:42.062786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062725    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47798c3a-363a-4a27-9148-ed8caf915e1a" containerName="payload-pre-processing"
Oct 30 14:05:42.062786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062740    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" containerName="payload-processing"
Oct 30 14:05:42.062786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062746    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" containerName="payload-processing"
Oct 30 14:05:42.062786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062755    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy"
Oct 30 14:05:42.062786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062760    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy"
Oct 30 14:05:42.062786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062781    4291 cpu_manager.go:414] "containerMap: removing container" podUID="3e5ac6a7-7352-42e4-8214-13f674589a0c" containerName="revoke-keys"
Oct 30 14:05:42.062786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062785    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e5ac6a7-7352-42e4-8214-13f674589a0c" containerName="revoke-keys"
Oct 30 14:05:42.063063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062801    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy"
Oct 30 14:05:42.063063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062806    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy"
Oct 30 14:05:42.063063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062891    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="47798c3a-363a-4a27-9148-ed8caf915e1a" containerName="payload-pre-processing"
Oct 30 14:05:42.063063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062915    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c272e255-8e03-4edb-8dd2-50b0fb50c19d" containerName="payload-processing"
Oct 30 14:05:42.063063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062924    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy"
Oct 30 14:05:42.063063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062934    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e5ac6a7-7352-42e4-8214-13f674589a0c" containerName="revoke-keys"
Oct 30 14:05:42.063063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.062941    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bafd4569-5128-4522-86f8-12b44c954428" containerName="istio-proxy"
Oct 30 14:05:42.068328 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.068307    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.082896 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.082863    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:05:42.083596 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.083568    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"]
Oct 30 14:05:42.090582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.090559    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:05:42.090768 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.090673    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:05:42.090768 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.090695    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:05:42.245813 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.245782    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.246007 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.245926    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.246070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.246038    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqk8g\" (UniqueName: \"kubernetes.io/projected/003dfff3-ae83-446f-8edc-2c1b73843e14-kube-api-access-jqk8g\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.347097 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.347062    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.347282 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.347158    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.347360 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.347307    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqk8g\" (UniqueName: \"kubernetes.io/projected/003dfff3-ae83-446f-8edc-2c1b73843e14-kube-api-access-jqk8g\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.347912 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.347886    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.349408 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.349387    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.366128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.366104    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqk8g\" (UniqueName: \"kubernetes.io/projected/003dfff3-ae83-446f-8edc-2c1b73843e14-kube-api-access-jqk8g\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.387055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.387029    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:42.721377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:42.721332    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"]
Oct 30 14:05:42.727973 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:05:42.725420    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod003dfff3_ae83_446f_8edc_2c1b73843e14.slice/crio-cdd636249f32269f4057f3feaa377a02f84f5625907e461116be6233f21bcfda WatchSource:0}: Error finding container cdd636249f32269f4057f3feaa377a02f84f5625907e461116be6233f21bcfda: Status 404 returned error can't find the container with id cdd636249f32269f4057f3feaa377a02f84f5625907e461116be6233f21bcfda
Oct 30 14:05:43.367624 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:43.367585    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp" event={"ID":"003dfff3-ae83-446f-8edc-2c1b73843e14","Type":"ContainerStarted","Data":"7d2b636a9e486f622dd0d65c8ffc703e385ab6afd39a9e26500c60b141499f11"}
Oct 30 14:05:43.367805 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:43.367667    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp" event={"ID":"003dfff3-ae83-446f-8edc-2c1b73843e14","Type":"ContainerStarted","Data":"cdd636249f32269f4057f3feaa377a02f84f5625907e461116be6233f21bcfda"}
Oct 30 14:05:43.408425 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:43.408376    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp" podStartSLOduration=1.408363518 podStartE2EDuration="1.408363518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:05:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:05:43.405750843 +0000 UTC m=+3753.665989105" watchObservedRunningTime="2026-10-30 14:05:43.408363518 +0000 UTC m=+3753.668601751"
Oct 30 14:05:44.376516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:44.376475    4291 generic.go:356] "Generic (PLEG): container finished" podID="003dfff3-ae83-446f-8edc-2c1b73843e14" containerID="7d2b636a9e486f622dd0d65c8ffc703e385ab6afd39a9e26500c60b141499f11" exitCode=0
Oct 30 14:05:44.376918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:44.376555    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp" event={"ID":"003dfff3-ae83-446f-8edc-2c1b73843e14","Type":"ContainerDied","Data":"7d2b636a9e486f622dd0d65c8ffc703e385ab6afd39a9e26500c60b141499f11"}
Oct 30 14:05:45.549823 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.549797    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:45.612840 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.612806    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/003dfff3-ae83-446f-8edc-2c1b73843e14-kube-api-access-jqk8g\" (UniqueName: \"kubernetes.io/projected/003dfff3-ae83-446f-8edc-2c1b73843e14-kube-api-access-jqk8g\") pod \"003dfff3-ae83-446f-8edc-2c1b73843e14\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") "
Oct 30 14:05:45.613055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.612877    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-openshift-ca\") pod \"003dfff3-ae83-446f-8edc-2c1b73843e14\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") "
Oct 30 14:05:45.613185 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.613146    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-xks-ca\") pod \"003dfff3-ae83-446f-8edc-2c1b73843e14\" (UID: \"003dfff3-ae83-446f-8edc-2c1b73843e14\") "
Oct 30 14:05:45.613962 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.613931    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-openshift-ca" pod "003dfff3-ae83-446f-8edc-2c1b73843e14" (UID: "003dfff3-ae83-446f-8edc-2c1b73843e14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:05:45.615557 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.615527    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/003dfff3-ae83-446f-8edc-2c1b73843e14-kube-api-access-jqk8g" pod "003dfff3-ae83-446f-8edc-2c1b73843e14" (UID: "003dfff3-ae83-446f-8edc-2c1b73843e14"). InnerVolumeSpecName "kube-api-access-jqk8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:45.615724 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.615697    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-xks-ca" pod "003dfff3-ae83-446f-8edc-2c1b73843e14" (UID: "003dfff3-ae83-446f-8edc-2c1b73843e14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:05:45.699816 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.699719    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:05:45.699816 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.699796    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:05:45.715406 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.715377    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:45.715406 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.715403    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqk8g\" (UniqueName: \"kubernetes.io/projected/003dfff3-ae83-446f-8edc-2c1b73843e14-kube-api-access-jqk8g\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:45.715564 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:45.715414    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/003dfff3-ae83-446f-8edc-2c1b73843e14-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.394603 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.394566    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"
Oct 30 14:05:46.394603 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.394582    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp" event={"ID":"003dfff3-ae83-446f-8edc-2c1b73843e14","Type":"ContainerDied","Data":"cdd636249f32269f4057f3feaa377a02f84f5625907e461116be6233f21bcfda"}
Oct 30 14:05:46.394883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.394684    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cdd636249f32269f4057f3feaa377a02f84f5625907e461116be6233f21bcfda"
Oct 30 14:05:46.768466 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.768436    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 14:05:46.849875 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.849824    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-certs\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.849875 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.849874    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-kube-api-access-nm48t\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-kube-api-access-nm48t\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.849905    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-data\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.849967    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba32c77-b382-4ea2-b087-785f9335d376-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba32c77-b382-4ea2-b087-785f9335d376-istiod-ca-cert\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850016    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-envoy\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850083    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-socket\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850140    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-istio-token\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-istio-token\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850475 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850174    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-certs" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:46.850475 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850218    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-credential-socket\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850475 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850312    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/aba32c77-b382-4ea2-b087-785f9335d376-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba32c77-b382-4ea2-b087-785f9335d376-istio-podinfo\") pod \"aba32c77-b382-4ea2-b087-785f9335d376\" (UID: \"aba32c77-b382-4ea2-b087-785f9335d376\") "
Oct 30 14:05:46.850475 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850398    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-socket" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:46.850732 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850496    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-data" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:46.850732 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850498    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba32c77-b382-4ea2-b087-785f9335d376-istiod-ca-cert" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:05:46.850732 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850525    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-credential-socket" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:46.850983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850958    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.851077 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.850990    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.851077 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.851005    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.851077 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.851018    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.851077 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.851030    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/aba32c77-b382-4ea2-b087-785f9335d376-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.852893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.852827    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/aba32c77-b382-4ea2-b087-785f9335d376-istio-podinfo" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:05:46.853019 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.852954    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-kube-api-access-nm48t" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "kube-api-access-nm48t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:46.853090 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.853010    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-envoy" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:05:46.853284 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.853258    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-istio-token" pod "aba32c77-b382-4ea2-b087-785f9335d376" (UID: "aba32c77-b382-4ea2-b087-785f9335d376"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:05:46.952470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.952320    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/aba32c77-b382-4ea2-b087-785f9335d376-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.952470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.952355    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.952470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.952371    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/aba32c77-b382-4ea2-b087-785f9335d376-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:46.952470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:46.952383    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nm48t\" (UniqueName: \"kubernetes.io/projected/aba32c77-b382-4ea2-b087-785f9335d376-kube-api-access-nm48t\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:05:47.403894 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.403857    4291 generic.go:356] "Generic (PLEG): container finished" podID="aba32c77-b382-4ea2-b087-785f9335d376" containerID="7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636" exitCode=0
Oct 30 14:05:47.404094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.403929    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"
Oct 30 14:05:47.404094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.403948    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" event={"ID":"aba32c77-b382-4ea2-b087-785f9335d376","Type":"ContainerDied","Data":"7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636"}
Oct 30 14:05:47.404094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.404012    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w" event={"ID":"aba32c77-b382-4ea2-b087-785f9335d376","Type":"ContainerDied","Data":"823d405353d0bd00354e81285950ade0e110b1485f2c44269405cc52e429b5ff"}
Oct 30 14:05:47.404094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.404044    4291 scope.go:122] "RemoveContainer" containerID="7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636"
Oct 30 14:05:47.417737 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.417702    4291 scope.go:122] "RemoveContainer" containerID="9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016"
Oct 30 14:05:47.429914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.429884    4291 scope.go:122] "RemoveContainer" containerID="7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636"
Oct 30 14:05:47.430318 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:05:47.430265    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636\": container with ID starting with 7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636 not found: ID does not exist" containerID="7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636"
Oct 30 14:05:47.430318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.430297    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636"} err="failed to get container status \"7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636\": rpc error: code = NotFound desc = could not find container \"7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636\": container with ID starting with 7d474d9cc14416af3bc3bcf6e757eb3a60c7cc88532a0d2b51332da773fec636 not found: ID does not exist"
Oct 30 14:05:47.430318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.430315    4291 scope.go:122] "RemoveContainer" containerID="9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016"
Oct 30 14:05:47.430627 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:05:47.430599    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016\": container with ID starting with 9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016 not found: ID does not exist" containerID="9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016"
Oct 30 14:05:47.430726 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.430665    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016"} err="failed to get container status \"9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016\": rpc error: code = NotFound desc = could not find container \"9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016\": container with ID starting with 9868756d642210e69fedf88f6fea3e97f89354c51e0b5bde1caf568ace291016 not found: ID does not exist"
Oct 30 14:05:47.456073 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.456041    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"]
Oct 30 14:05:47.467659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.467607    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-e3eb98-openshift-default-69f94dc64f-mq22w"]
Oct 30 14:05:47.881498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:05:47.881443    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba32c77-b382-4ea2-b087-785f9335d376" path="/var/lib/kubelet/pods/aba32c77-b382-4ea2-b087-785f9335d376/volumes"
Oct 30 14:06:07.627624 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:07.627588    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:06:08.211623 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:06:08.211585    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:06:14.293132 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:14.293099    4291 scope.go:122] "RemoveContainer" containerID="a200e87452b311259ec70a2fc20fa0a2857a75dd9a5eabdea15274ca92b765b8"
Oct 30 14:06:14.304210 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:14.304188    4291 scope.go:122] "RemoveContainer" containerID="951849d9c41948f9ceabe8e924a6fd61803ecc4cce2c65ba9468383a84a19443"
Oct 30 14:06:14.315077 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:14.315043    4291 scope.go:122] "RemoveContainer" containerID="239a4eb03ab693a0a83be255bfa377a14ff8931ebf4f72a74aa4f0358cf9747f"
Oct 30 14:06:18.311441 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.311394    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"]
Oct 30 14:06:18.312245 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.312220    4291 cpu_manager.go:414] "containerMap: removing container" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy"
Oct 30 14:06:18.312357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.312245    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy"
Oct 30 14:06:18.312357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.312274    4291 cpu_manager.go:414] "containerMap: removing container" podUID="003dfff3-ae83-446f-8edc-2c1b73843e14" containerName="revoke-keys"
Oct 30 14:06:18.312357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.312283    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="003dfff3-ae83-446f-8edc-2c1b73843e14" containerName="revoke-keys"
Oct 30 14:06:18.312535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.312495    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy"
Oct 30 14:06:18.312535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.312513    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="003dfff3-ae83-446f-8edc-2c1b73843e14" containerName="revoke-keys"
Oct 30 14:06:18.312535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.312530    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy"
Oct 30 14:06:18.316150 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.316128    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.331827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.331793    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"]
Oct 30 14:06:18.339669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.339628    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:06:18.339841 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.339652    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:06:18.339904 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.339685    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:06:18.341793 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.341739    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:06:18.450747 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.450705    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.450947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.450842    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.450947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.450884    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmmb2\" (UniqueName: \"kubernetes.io/projected/7c1a747a-3b92-4bc7-81f3-4848428da4db-kube-api-access-fmmb2\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.552282 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.552239    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmmb2\" (UniqueName: \"kubernetes.io/projected/7c1a747a-3b92-4bc7-81f3-4848428da4db-kube-api-access-fmmb2\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.552514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.552417    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.552514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.552475    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.553151 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.553125    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.554871 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.554848    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.571414 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.571344    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmmb2\" (UniqueName: \"kubernetes.io/projected/7c1a747a-3b92-4bc7-81f3-4848428da4db-kube-api-access-fmmb2\") pod \"maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.631138 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.631097    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:18.942196 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:18.942169    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"]
Oct 30 14:06:18.944371 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:06:18.944325    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c1a747a_3b92_4bc7_81f3_4848428da4db.slice/crio-28489f84ca9e577d936c506489aafbbdf112d9f0c3f979c8e1d18885482c6e1b WatchSource:0}: Error finding container 28489f84ca9e577d936c506489aafbbdf112d9f0c3f979c8e1d18885482c6e1b: Status 404 returned error can't find the container with id 28489f84ca9e577d936c506489aafbbdf112d9f0c3f979c8e1d18885482c6e1b
Oct 30 14:06:19.730016 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:19.729980    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99" event={"ID":"7c1a747a-3b92-4bc7-81f3-4848428da4db","Type":"ContainerStarted","Data":"01df92db71371080e2f434d9a6ce5fd48a5d5aa240b8ea61e9985167c5cb9c9f"}
Oct 30 14:06:19.730447 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:19.730035    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99" event={"ID":"7c1a747a-3b92-4bc7-81f3-4848428da4db","Type":"ContainerStarted","Data":"28489f84ca9e577d936c506489aafbbdf112d9f0c3f979c8e1d18885482c6e1b"}
Oct 30 14:06:19.768143 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:19.768064    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99" podStartSLOduration=1.768048881 podStartE2EDuration="1.768048881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:19.766575181 +0000 UTC m=+3790.026813415" watchObservedRunningTime="2026-10-30 14:06:19.768048881 +0000 UTC m=+3790.028287115"
Oct 30 14:06:20.738391 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:20.738353    4291 generic.go:356] "Generic (PLEG): container finished" podID="7c1a747a-3b92-4bc7-81f3-4848428da4db" containerID="01df92db71371080e2f434d9a6ce5fd48a5d5aa240b8ea61e9985167c5cb9c9f" exitCode=0
Oct 30 14:06:20.738842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:20.738442    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99" event={"ID":"7c1a747a-3b92-4bc7-81f3-4848428da4db","Type":"ContainerDied","Data":"01df92db71371080e2f434d9a6ce5fd48a5d5aa240b8ea61e9985167c5cb9c9f"}
Oct 30 14:06:21.922359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:21.922335    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:22.014839 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.014744    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-xks-ca\") pod \"7c1a747a-3b92-4bc7-81f3-4848428da4db\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") "
Oct 30 14:06:22.015013 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.014887    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1a747a-3b92-4bc7-81f3-4848428da4db-kube-api-access-fmmb2\" (UniqueName: \"kubernetes.io/projected/7c1a747a-3b92-4bc7-81f3-4848428da4db-kube-api-access-fmmb2\") pod \"7c1a747a-3b92-4bc7-81f3-4848428da4db\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") "
Oct 30 14:06:22.015013 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.014960    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-openshift-ca\") pod \"7c1a747a-3b92-4bc7-81f3-4848428da4db\" (UID: \"7c1a747a-3b92-4bc7-81f3-4848428da4db\") "
Oct 30 14:06:22.015328 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.015299    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-openshift-ca" pod "7c1a747a-3b92-4bc7-81f3-4848428da4db" (UID: "7c1a747a-3b92-4bc7-81f3-4848428da4db"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:22.015729 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.015703    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:22.017161 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.017136    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-xks-ca" pod "7c1a747a-3b92-4bc7-81f3-4848428da4db" (UID: "7c1a747a-3b92-4bc7-81f3-4848428da4db"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:22.017364 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.017338    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1a747a-3b92-4bc7-81f3-4848428da4db-kube-api-access-fmmb2" pod "7c1a747a-3b92-4bc7-81f3-4848428da4db" (UID: "7c1a747a-3b92-4bc7-81f3-4848428da4db"). InnerVolumeSpecName "kube-api-access-fmmb2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:22.116776 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.116742    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7c1a747a-3b92-4bc7-81f3-4848428da4db-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:22.116776 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.116772    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmmb2\" (UniqueName: \"kubernetes.io/projected/7c1a747a-3b92-4bc7-81f3-4848428da4db-kube-api-access-fmmb2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:22.754914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.754885    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"
Oct 30 14:06:22.755107 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.754879    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99" event={"ID":"7c1a747a-3b92-4bc7-81f3-4848428da4db","Type":"ContainerDied","Data":"28489f84ca9e577d936c506489aafbbdf112d9f0c3f979c8e1d18885482c6e1b"}
Oct 30 14:06:22.755107 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:22.754991    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28489f84ca9e577d936c506489aafbbdf112d9f0c3f979c8e1d18885482c6e1b"
Oct 30 14:06:28.742374 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:28.742333    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"]
Oct 30 14:06:31.329153 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:31.329112    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"]
Oct 30 14:06:31.391318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:31.378358    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-cab6c5-8zcvp"]
Oct 30 14:06:31.879399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:31.879360    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="003dfff3-ae83-446f-8edc-2c1b73843e14" path="/var/lib/kubelet/pods/003dfff3-ae83-446f-8edc-2c1b73843e14/volumes"
Oct 30 14:06:33.742670 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:33.742600    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" podUID="ed7440e2-c465-496c-8172-161a37301ab9" containerName="payload-pre-processing" containerID="cri-o://5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93" gracePeriod=25
Oct 30 14:06:37.098049 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:37.098006    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" podUID="ed7440e2-c465-496c-8172-161a37301ab9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:06:38.283156 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:06:38.283114    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:06:39.042877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.042840    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"]
Oct 30 14:06:39.043491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.043469    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7c1a747a-3b92-4bc7-81f3-4848428da4db" containerName="revoke-keys"
Oct 30 14:06:39.043606 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.043492    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1a747a-3b92-4bc7-81f3-4848428da4db" containerName="revoke-keys"
Oct 30 14:06:39.043606 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.043514    4291 cpu_manager.go:414] "containerMap: removing container" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy"
Oct 30 14:06:39.043606 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.043521    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba32c77-b382-4ea2-b087-785f9335d376" containerName="istio-proxy"
Oct 30 14:06:39.043848 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.043728    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1a747a-3b92-4bc7-81f3-4848428da4db" containerName="revoke-keys"
Oct 30 14:06:39.048821 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.048797    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.071494 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.071467    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:06:39.071667 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.071586    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:06:39.083611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.083588    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:06:39.083825 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.083673    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:06:39.089253 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.089229    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"]
Oct 30 14:06:39.109082 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.109053    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.109263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.109242    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssqlc\" (UniqueName: \"kubernetes.io/projected/34f9e406-58ec-4170-bd8c-73a4ce013576-kube-api-access-ssqlc\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.109363 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.109284    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.211162 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.211124    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.211353 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.211334    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssqlc\" (UniqueName: \"kubernetes.io/projected/34f9e406-58ec-4170-bd8c-73a4ce013576-kube-api-access-ssqlc\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.211431 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.211376    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.212218 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.212188    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.213854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.213825    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.236175 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.236147    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssqlc\" (UniqueName: \"kubernetes.io/projected/34f9e406-58ec-4170-bd8c-73a4ce013576-kube-api-access-ssqlc\") pod \"maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.363658 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.363545    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:39.757762 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.757730    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"]
Oct 30 14:06:39.759211 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:06:39.759185    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34f9e406_58ec_4170_bd8c_73a4ce013576.slice/crio-cb8ec576fa2eae5d26442b80ee4771d14cc61d3da7796a0cf7a55c3823f717cb WatchSource:0}: Error finding container cb8ec576fa2eae5d26442b80ee4771d14cc61d3da7796a0cf7a55c3823f717cb: Status 404 returned error can't find the container with id cb8ec576fa2eae5d26442b80ee4771d14cc61d3da7796a0cf7a55c3823f717cb
Oct 30 14:06:39.824693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.824624    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"]
Oct 30 14:06:39.830378 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.830359    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:39.846236 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.846209    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"]
Oct 30 14:06:39.899190 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.899155    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz" event={"ID":"34f9e406-58ec-4170-bd8c-73a4ce013576","Type":"ContainerStarted","Data":"cb8ec576fa2eae5d26442b80ee4771d14cc61d3da7796a0cf7a55c3823f717cb"}
Oct 30 14:06:39.929977 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.929943    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b464k\" (UniqueName: \"kubernetes.io/projected/5e6f9208-f667-4675-8c18-f4fa9ef572a5-kube-api-access-b464k\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:39.930167 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.930075    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:39.930244 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:39.930223    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.036117 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.032823    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b464k\" (UniqueName: \"kubernetes.io/projected/5e6f9208-f667-4675-8c18-f4fa9ef572a5-kube-api-access-b464k\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.036117 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.032952    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.036117 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.033123    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.046672 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.040723    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.046672 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.042107    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.067040 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.066954    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b464k\" (UniqueName: \"kubernetes.io/projected/5e6f9208-f667-4675-8c18-f4fa9ef572a5-kube-api-access-b464k\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.145814 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.145779    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:40.460524 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.460484    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"]
Oct 30 14:06:40.907660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.907593    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df" event={"ID":"5e6f9208-f667-4675-8c18-f4fa9ef572a5","Type":"ContainerStarted","Data":"4ef22b89322654ffd28ef638a957c37bdce9465d6f656e9d30e7ef93faaea5ea"}
Oct 30 14:06:40.907660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.907662    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df" event={"ID":"5e6f9208-f667-4675-8c18-f4fa9ef572a5","Type":"ContainerStarted","Data":"a21f09f254f1b93a225f642519a6e9e85906e8de0350d0ed3dba782c768742be"}
Oct 30 14:06:40.909367 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.909328    4291 generic.go:356] "Generic (PLEG): container finished" podID="34f9e406-58ec-4170-bd8c-73a4ce013576" containerID="f94d84ad5546c6bb3f4a93dd604ebc4d5b5030dde1a22323f778cf6cf3ce1967" exitCode=0
Oct 30 14:06:40.909508 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.909379    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz" event={"ID":"34f9e406-58ec-4170-bd8c-73a4ce013576","Type":"ContainerDied","Data":"f94d84ad5546c6bb3f4a93dd604ebc4d5b5030dde1a22323f778cf6cf3ce1967"}
Oct 30 14:06:40.944856 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:40.944803    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df" podStartSLOduration=1.944789342 podStartE2EDuration="1.944789342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:40.944018248 +0000 UTC m=+3811.204256482" watchObservedRunningTime="2026-10-30 14:06:40.944789342 +0000 UTC m=+3811.205027576"
Oct 30 14:06:42.068670 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.068296    4291 generic.go:356] "Generic (PLEG): container finished" podID="5e6f9208-f667-4675-8c18-f4fa9ef572a5" containerID="4ef22b89322654ffd28ef638a957c37bdce9465d6f656e9d30e7ef93faaea5ea" exitCode=0
Oct 30 14:06:42.074131 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.068364    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df" event={"ID":"5e6f9208-f667-4675-8c18-f4fa9ef572a5","Type":"ContainerDied","Data":"4ef22b89322654ffd28ef638a957c37bdce9465d6f656e9d30e7ef93faaea5ea"}
Oct 30 14:06:42.252173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.252145    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"]
Oct 30 14:06:42.255585 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.255555    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-bd2fed-kgs99"]
Oct 30 14:06:42.265291 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.265269    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:42.401657 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.389030    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34f9e406-58ec-4170-bd8c-73a4ce013576-kube-api-access-ssqlc\" (UniqueName: \"kubernetes.io/projected/34f9e406-58ec-4170-bd8c-73a4ce013576-kube-api-access-ssqlc\") pod \"34f9e406-58ec-4170-bd8c-73a4ce013576\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") "
Oct 30 14:06:42.401657 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.389116    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-openshift-ca\") pod \"34f9e406-58ec-4170-bd8c-73a4ce013576\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") "
Oct 30 14:06:42.401657 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.389306    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-xks-ca\") pod \"34f9e406-58ec-4170-bd8c-73a4ce013576\" (UID: \"34f9e406-58ec-4170-bd8c-73a4ce013576\") "
Oct 30 14:06:42.414228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.414194    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-openshift-ca" pod "34f9e406-58ec-4170-bd8c-73a4ce013576" (UID: "34f9e406-58ec-4170-bd8c-73a4ce013576"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:42.417851 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.417823    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34f9e406-58ec-4170-bd8c-73a4ce013576-kube-api-access-ssqlc" pod "34f9e406-58ec-4170-bd8c-73a4ce013576" (UID: "34f9e406-58ec-4170-bd8c-73a4ce013576"). InnerVolumeSpecName "kube-api-access-ssqlc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:42.427605 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.427580    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-xks-ca" pod "34f9e406-58ec-4170-bd8c-73a4ce013576" (UID: "34f9e406-58ec-4170-bd8c-73a4ce013576"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:42.491582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.491541    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssqlc\" (UniqueName: \"kubernetes.io/projected/34f9e406-58ec-4170-bd8c-73a4ce013576-kube-api-access-ssqlc\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:42.491582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.491581    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:42.491811 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:42.491599    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34f9e406-58ec-4170-bd8c-73a4ce013576-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:43.045662 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.039864    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"]
Oct 30 14:06:43.045662 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.045281    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73188bb4-jxch5"]
Oct 30 14:06:43.094517 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.094111    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"
Oct 30 14:06:43.113148 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.109452    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz" event={"ID":"34f9e406-58ec-4170-bd8c-73a4ce013576","Type":"ContainerDied","Data":"cb8ec576fa2eae5d26442b80ee4771d14cc61d3da7796a0cf7a55c3823f717cb"}
Oct 30 14:06:43.113148 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.109528    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb8ec576fa2eae5d26442b80ee4771d14cc61d3da7796a0cf7a55c3823f717cb"
Oct 30 14:06:43.287138 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.287110    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:43.412561 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.412533    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6f9208-f667-4675-8c18-f4fa9ef572a5-kube-api-access-b464k\" (UniqueName: \"kubernetes.io/projected/5e6f9208-f667-4675-8c18-f4fa9ef572a5-kube-api-access-b464k\") pod \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") "
Oct 30 14:06:43.412762 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.412614    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-openshift-ca\") pod \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") "
Oct 30 14:06:43.412815 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.412782    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-xks-ca\") pod \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\" (UID: \"5e6f9208-f667-4675-8c18-f4fa9ef572a5\") "
Oct 30 14:06:43.412980 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.412953    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-openshift-ca" pod "5e6f9208-f667-4675-8c18-f4fa9ef572a5" (UID: "5e6f9208-f667-4675-8c18-f4fa9ef572a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:43.413244 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.413230    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:43.414720 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.414692    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6f9208-f667-4675-8c18-f4fa9ef572a5-kube-api-access-b464k" pod "5e6f9208-f667-4675-8c18-f4fa9ef572a5" (UID: "5e6f9208-f667-4675-8c18-f4fa9ef572a5"). InnerVolumeSpecName "kube-api-access-b464k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:43.414826 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.414814    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-xks-ca" pod "5e6f9208-f667-4675-8c18-f4fa9ef572a5" (UID: "5e6f9208-f667-4675-8c18-f4fa9ef572a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:43.514929 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.514899    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b464k\" (UniqueName: \"kubernetes.io/projected/5e6f9208-f667-4675-8c18-f4fa9ef572a5-kube-api-access-b464k\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:43.514929 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.514928    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6f9208-f667-4675-8c18-f4fa9ef572a5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:43.879670 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.879621    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1a747a-3b92-4bc7-81f3-4848428da4db" path="/var/lib/kubelet/pods/7c1a747a-3b92-4bc7-81f3-4848428da4db/volumes"
Oct 30 14:06:43.880193 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:43.880177    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f672a605-4281-49a2-8bd9-b3806eb18578" path="/var/lib/kubelet/pods/f672a605-4281-49a2-8bd9-b3806eb18578/volumes"
Oct 30 14:06:44.103530 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:44.103494    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df" event={"ID":"5e6f9208-f667-4675-8c18-f4fa9ef572a5","Type":"ContainerDied","Data":"a21f09f254f1b93a225f642519a6e9e85906e8de0350d0ed3dba782c768742be"}
Oct 30 14:06:44.104091 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:44.103550    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a21f09f254f1b93a225f642519a6e9e85906e8de0350d0ed3dba782c768742be"
Oct 30 14:06:44.104091 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:44.103504    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"
Oct 30 14:06:46.353295 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:46.353261    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:06:47.099051 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:47.099008    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" podUID="ed7440e2-c465-496c-8172-161a37301ab9" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:06:50.784210 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.784162    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"]
Oct 30 14:06:50.785015 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.784992    4291 cpu_manager.go:414] "containerMap: removing container" podUID="5e6f9208-f667-4675-8c18-f4fa9ef572a5" containerName="revoke-keys"
Oct 30 14:06:50.785015 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.785014    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6f9208-f667-4675-8c18-f4fa9ef572a5" containerName="revoke-keys"
Oct 30 14:06:50.785198 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.785069    4291 cpu_manager.go:414] "containerMap: removing container" podUID="34f9e406-58ec-4170-bd8c-73a4ce013576" containerName="revoke-keys"
Oct 30 14:06:50.785198 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.785077    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34f9e406-58ec-4170-bd8c-73a4ce013576" containerName="revoke-keys"
Oct 30 14:06:50.785308 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.785244    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6f9208-f667-4675-8c18-f4fa9ef572a5" containerName="revoke-keys"
Oct 30 14:06:50.785308 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.785268    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="34f9e406-58ec-4170-bd8c-73a4ce013576" containerName="revoke-keys"
Oct 30 14:06:50.788998 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.788972    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.834669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.829699    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:06:50.834669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.829823    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:06:50.834669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.829954    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:06:50.834669 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.830075    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:06:50.871658 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.870762    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.871658 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.870841    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.871658 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.870908    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpdzl\" (UniqueName: \"kubernetes.io/projected/569d1268-67e4-4ff7-a710-9183c91fdc93-kube-api-access-gpdzl\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.884657 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.881257    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"]
Oct 30 14:06:50.981109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.972500    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.981109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.974205    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.981109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.974298    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpdzl\" (UniqueName: \"kubernetes.io/projected/569d1268-67e4-4ff7-a710-9183c91fdc93-kube-api-access-gpdzl\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.981109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.976182    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.981109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.979105    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:50.995324 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:50.995276    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpdzl\" (UniqueName: \"kubernetes.io/projected/569d1268-67e4-4ff7-a710-9183c91fdc93-kube-api-access-gpdzl\") pod \"maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:51.147649 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:51.147611    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:51.499809 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:51.499329    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"]
Oct 30 14:06:52.163149 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:52.163109    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h" event={"ID":"569d1268-67e4-4ff7-a710-9183c91fdc93","Type":"ContainerStarted","Data":"55bbf5b6bda6393808fd4698664f749c89060dd114ef49b49de00530641a1191"}
Oct 30 14:06:52.163617 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:52.163164    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h" event={"ID":"569d1268-67e4-4ff7-a710-9183c91fdc93","Type":"ContainerStarted","Data":"d87b39006d4c2d1b1c24b1607617f281ead273eee2ae953fc2ed9327993a1707"}
Oct 30 14:06:52.205254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:52.205199    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h" podStartSLOduration=2.205185168 podStartE2EDuration="2.205185168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:06:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:06:52.201715449 +0000 UTC m=+3822.461953683" watchObservedRunningTime="2026-10-30 14:06:52.205185168 +0000 UTC m=+3822.465423406"
Oct 30 14:06:53.172039 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:53.172003    4291 generic.go:356] "Generic (PLEG): container finished" podID="569d1268-67e4-4ff7-a710-9183c91fdc93" containerID="55bbf5b6bda6393808fd4698664f749c89060dd114ef49b49de00530641a1191" exitCode=0
Oct 30 14:06:53.172448 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:53.172092    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h" event={"ID":"569d1268-67e4-4ff7-a710-9183c91fdc93","Type":"ContainerDied","Data":"55bbf5b6bda6393808fd4698664f749c89060dd114ef49b49de00530641a1191"}
Oct 30 14:06:54.017610 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.017538    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 14:06:54.116017 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.115979    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed7440e2-c465-496c-8172-161a37301ab9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7440e2-c465-496c-8172-161a37301ab9-plugins-config-volume\") pod \"ed7440e2-c465-496c-8172-161a37301ab9\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") "
Oct 30 14:06:54.116017 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.116020    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed7440e2-c465-496c-8172-161a37301ab9-kube-api-access-v5gk5\" (UniqueName: \"kubernetes.io/projected/ed7440e2-c465-496c-8172-161a37301ab9-kube-api-access-v5gk5\") pod \"ed7440e2-c465-496c-8172-161a37301ab9\" (UID: \"ed7440e2-c465-496c-8172-161a37301ab9\") "
Oct 30 14:06:54.116457 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.116430    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed7440e2-c465-496c-8172-161a37301ab9-plugins-config-volume" pod "ed7440e2-c465-496c-8172-161a37301ab9" (UID: "ed7440e2-c465-496c-8172-161a37301ab9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:54.118375 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.118352    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed7440e2-c465-496c-8172-161a37301ab9-kube-api-access-v5gk5" pod "ed7440e2-c465-496c-8172-161a37301ab9" (UID: "ed7440e2-c465-496c-8172-161a37301ab9"). InnerVolumeSpecName "kube-api-access-v5gk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:54.181981 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.181949    4291 generic.go:356] "Generic (PLEG): container finished" podID="ed7440e2-c465-496c-8172-161a37301ab9" containerID="5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93" exitCode=0
Oct 30 14:06:54.182473 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.181995    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"
Oct 30 14:06:54.182473 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.182032    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" event={"ID":"ed7440e2-c465-496c-8172-161a37301ab9","Type":"ContainerDied","Data":"5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93"}
Oct 30 14:06:54.182473 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.182094    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z" event={"ID":"ed7440e2-c465-496c-8172-161a37301ab9","Type":"ContainerDied","Data":"bb8dc626a4aea418a9af099521cf5fca5cd5bb61b1f6b5b56de5ae9048d1cd86"}
Oct 30 14:06:54.182473 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.182126    4291 scope.go:122] "RemoveContainer" containerID="5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93"
Oct 30 14:06:54.193515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.193494    4291 scope.go:122] "RemoveContainer" containerID="5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93"
Oct 30 14:06:54.193854 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:06:54.193830    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93\": container with ID starting with 5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93 not found: ID does not exist" containerID="5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93"
Oct 30 14:06:54.193966 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.193860    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93"} err="failed to get container status \"5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93\": rpc error: code = NotFound desc = could not find container \"5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93\": container with ID starting with 5da1c6e1789ff4f8de037427b7273b33261ad51aba3b15a5224d26ebb667da93 not found: ID does not exist"
Oct 30 14:06:54.218496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.218439    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed7440e2-c465-496c-8172-161a37301ab9-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.218496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.218496    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5gk5\" (UniqueName: \"kubernetes.io/projected/ed7440e2-c465-496c-8172-161a37301ab9-kube-api-access-v5gk5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.232598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.232564    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"]
Oct 30 14:06:54.243073 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.243044    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-bd2fed-777cf5c495-n265z"]
Oct 30 14:06:54.342953 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.342929    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:54.420824 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.420786    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-openshift-ca\") pod \"569d1268-67e4-4ff7-a710-9183c91fdc93\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") "
Oct 30 14:06:54.421004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.420898    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-xks-ca\") pod \"569d1268-67e4-4ff7-a710-9183c91fdc93\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") "
Oct 30 14:06:54.421004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.420946    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/569d1268-67e4-4ff7-a710-9183c91fdc93-kube-api-access-gpdzl\" (UniqueName: \"kubernetes.io/projected/569d1268-67e4-4ff7-a710-9183c91fdc93-kube-api-access-gpdzl\") pod \"569d1268-67e4-4ff7-a710-9183c91fdc93\" (UID: \"569d1268-67e4-4ff7-a710-9183c91fdc93\") "
Oct 30 14:06:54.421225 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.421196    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-openshift-ca" pod "569d1268-67e4-4ff7-a710-9183c91fdc93" (UID: "569d1268-67e4-4ff7-a710-9183c91fdc93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:06:54.421484 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.421459    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.423180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.423157    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-xks-ca" pod "569d1268-67e4-4ff7-a710-9183c91fdc93" (UID: "569d1268-67e4-4ff7-a710-9183c91fdc93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:06:54.423554 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.423536    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569d1268-67e4-4ff7-a710-9183c91fdc93-kube-api-access-gpdzl" pod "569d1268-67e4-4ff7-a710-9183c91fdc93" (UID: "569d1268-67e4-4ff7-a710-9183c91fdc93"). InnerVolumeSpecName "kube-api-access-gpdzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:06:54.523208 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.523174    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/569d1268-67e4-4ff7-a710-9183c91fdc93-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:54.523208 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:54.523203    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpdzl\" (UniqueName: \"kubernetes.io/projected/569d1268-67e4-4ff7-a710-9183c91fdc93-kube-api-access-gpdzl\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:06:55.190223 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:55.190190    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"
Oct 30 14:06:55.190691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:55.190228    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h" event={"ID":"569d1268-67e4-4ff7-a710-9183c91fdc93","Type":"ContainerDied","Data":"d87b39006d4c2d1b1c24b1607617f281ead273eee2ae953fc2ed9327993a1707"}
Oct 30 14:06:55.190691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:55.190279    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d87b39006d4c2d1b1c24b1607617f281ead273eee2ae953fc2ed9327993a1707"
Oct 30 14:06:55.880841 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:06:55.880797    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed7440e2-c465-496c-8172-161a37301ab9" path="/var/lib/kubelet/pods/ed7440e2-c465-496c-8172-161a37301ab9/volumes"
Oct 30 14:07:01.186732 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:01.186691    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"]
Oct 30 14:07:01.220094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:01.220050    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"]
Oct 30 14:07:06.192659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:06.187120    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" podUID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" containerName="payload-processing" containerID="cri-o://263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc" gracePeriod=25
Oct 30 14:07:06.224558 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:06.222855    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" podUID="52fda850-fa06-40c9-914e-0ad7ecde464c" containerName="payload-pre-processing" containerID="cri-o://bbb555928d5525c3d395b64217abcb702e2efbf238bac0131ee4f785f7c7f701" gracePeriod=25
Oct 30 14:07:08.210843 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:07:08.210802    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:07:09.707671 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:09.707605    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" podUID="52fda850-fa06-40c9-914e-0ad7ecde464c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:07:09.707671 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:09.707626    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" podUID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:07:09.816379 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:09.816338    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"]
Oct 30 14:07:09.829904 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:09.829865    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9b32c32a-nqjvz"]
Oct 30 14:07:09.906011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:09.905970    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34f9e406-58ec-4170-bd8c-73a4ce013576" path="/var/lib/kubelet/pods/34f9e406-58ec-4170-bd8c-73a4ce013576/volumes"
Oct 30 14:07:14.368882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:14.368850    4291 scope.go:122] "RemoveContainer" containerID="cb2896653baaed9f30fabd4cf8dd853ecf6619ca49327d700350630dec68bf4d"
Oct 30 14:07:14.379175 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:14.379153    4291 scope.go:122] "RemoveContainer" containerID="b01e29c6c37e0594fadb6a5fd0694135a4d19a32604c90fedae46d1fdfea5cfd"
Oct 30 14:07:14.388263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:14.388233    4291 scope.go:122] "RemoveContainer" containerID="0c8570be346c73c27282e8dc9c730f61690064627241112df82d67d99e0f0a23"
Oct 30 14:07:14.397014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:14.396991    4291 scope.go:122] "RemoveContainer" containerID="6208fa556af864c6da9be1032df30d8d0aa5dfb051fb800faf79222e47ceca5c"
Oct 30 14:07:19.746786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:19.746742    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" podUID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:07:19.754775 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:19.754716    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" podUID="52fda850-fa06-40c9-914e-0ad7ecde464c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:07:20.259268 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:20.259228    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"]
Oct 30 14:07:20.264125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:20.264096    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-552cc3-589df"]
Oct 30 14:07:21.879573 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:21.879541    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6f9208-f667-4675-8c18-f4fa9ef572a5" path="/var/lib/kubelet/pods/5e6f9208-f667-4675-8c18-f4fa9ef572a5/volumes"
Oct 30 14:07:23.092153 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:23.092114    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:07:26.518582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.518552    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 14:07:26.548443 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.548399    4291 generic.go:356] "Generic (PLEG): container finished" podID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" containerID="263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc" exitCode=0
Oct 30 14:07:26.548443 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.548438    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"
Oct 30 14:07:26.548777 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.548534    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" event={"ID":"67fb9d16-7c3e-4a74-93b6-0cede5462c07","Type":"ContainerDied","Data":"263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc"}
Oct 30 14:07:26.548777 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.548584    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6" event={"ID":"67fb9d16-7c3e-4a74-93b6-0cede5462c07","Type":"ContainerDied","Data":"b8a379b79a5b5a0da0c082b76c9320d45ac9f1e828b80531114b273aab27dfde"}
Oct 30 14:07:26.548777 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.548615    4291 scope.go:122] "RemoveContainer" containerID="263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc"
Oct 30 14:07:26.549867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.549845    4291 generic.go:356] "Generic (PLEG): container finished" podID="52fda850-fa06-40c9-914e-0ad7ecde464c" containerID="bbb555928d5525c3d395b64217abcb702e2efbf238bac0131ee4f785f7c7f701" exitCode=0
Oct 30 14:07:26.549963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.549869    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" event={"ID":"52fda850-fa06-40c9-914e-0ad7ecde464c","Type":"ContainerDied","Data":"bbb555928d5525c3d395b64217abcb702e2efbf238bac0131ee4f785f7c7f701"}
Oct 30 14:07:26.549963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.549912    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972" event={"ID":"52fda850-fa06-40c9-914e-0ad7ecde464c","Type":"ContainerDied","Data":"6e8a4804a72ed64a2aadc9d2c140cce75cb88cb5c09fe3ed3e8c25c4bc031f24"}
Oct 30 14:07:26.549963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.549928    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e8a4804a72ed64a2aadc9d2c140cce75cb88cb5c09fe3ed3e8c25c4bc031f24"
Oct 30 14:07:26.550892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.550873    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 14:07:26.556434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.556408    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67fb9d16-7c3e-4a74-93b6-0cede5462c07-kube-api-access-lmjdn\" (UniqueName: \"kubernetes.io/projected/67fb9d16-7c3e-4a74-93b6-0cede5462c07-kube-api-access-lmjdn\") pod \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") "
Oct 30 14:07:26.556611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.556549    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67fb9d16-7c3e-4a74-93b6-0cede5462c07-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fb9d16-7c3e-4a74-93b6-0cede5462c07-plugins-config-volume\") pod \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\" (UID: \"67fb9d16-7c3e-4a74-93b6-0cede5462c07\") "
Oct 30 14:07:26.557842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.557815    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67fb9d16-7c3e-4a74-93b6-0cede5462c07-plugins-config-volume" pod "67fb9d16-7c3e-4a74-93b6-0cede5462c07" (UID: "67fb9d16-7c3e-4a74-93b6-0cede5462c07"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:26.560388 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.560363    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67fb9d16-7c3e-4a74-93b6-0cede5462c07-kube-api-access-lmjdn" pod "67fb9d16-7c3e-4a74-93b6-0cede5462c07" (UID: "67fb9d16-7c3e-4a74-93b6-0cede5462c07"). InnerVolumeSpecName "kube-api-access-lmjdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:26.562823 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.562804    4291 scope.go:122] "RemoveContainer" containerID="263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc"
Oct 30 14:07:26.563139 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:07:26.563115    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc\": container with ID starting with 263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc not found: ID does not exist" containerID="263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc"
Oct 30 14:07:26.563245 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.563144    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc"} err="failed to get container status \"263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc\": rpc error: code = NotFound desc = could not find container \"263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc\": container with ID starting with 263d9a0e1c94a52a44ebab206a367359fa4d6954c399d81f993fdc2ffaf49ecc not found: ID does not exist"
Oct 30 14:07:26.658282 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.658246    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fda850-fa06-40c9-914e-0ad7ecde464c-kube-api-access-9n8l2\" (UniqueName: \"kubernetes.io/projected/52fda850-fa06-40c9-914e-0ad7ecde464c-kube-api-access-9n8l2\") pod \"52fda850-fa06-40c9-914e-0ad7ecde464c\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") "
Oct 30 14:07:26.658456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.658374    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52fda850-fa06-40c9-914e-0ad7ecde464c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda850-fa06-40c9-914e-0ad7ecde464c-plugins-config-volume\") pod \"52fda850-fa06-40c9-914e-0ad7ecde464c\" (UID: \"52fda850-fa06-40c9-914e-0ad7ecde464c\") "
Oct 30 14:07:26.658838 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.658812    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fda850-fa06-40c9-914e-0ad7ecde464c-plugins-config-volume" pod "52fda850-fa06-40c9-914e-0ad7ecde464c" (UID: "52fda850-fa06-40c9-914e-0ad7ecde464c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:26.659039 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.659022    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52fda850-fa06-40c9-914e-0ad7ecde464c-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:26.659078 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.659042    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmjdn\" (UniqueName: \"kubernetes.io/projected/67fb9d16-7c3e-4a74-93b6-0cede5462c07-kube-api-access-lmjdn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:26.659078 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.659063    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67fb9d16-7c3e-4a74-93b6-0cede5462c07-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:26.660372 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.660350    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fda850-fa06-40c9-914e-0ad7ecde464c-kube-api-access-9n8l2" pod "52fda850-fa06-40c9-914e-0ad7ecde464c" (UID: "52fda850-fa06-40c9-914e-0ad7ecde464c"). InnerVolumeSpecName "kube-api-access-9n8l2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:26.760105 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.760018    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9n8l2\" (UniqueName: \"kubernetes.io/projected/52fda850-fa06-40c9-914e-0ad7ecde464c-kube-api-access-9n8l2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:26.895258 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.895223    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"]
Oct 30 14:07:26.907234 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:26.907196    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-d2cecb-59cdc6ccd8-zhnx6"]
Oct 30 14:07:27.607964 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.607927    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"
Oct 30 14:07:27.663920 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.663884    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"]
Oct 30 14:07:27.666925 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.666894    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-d2cecb-n5n2h"]
Oct 30 14:07:27.672434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.672408    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"]
Oct 30 14:07:27.676904 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.676880    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-d2cecb-7f845b8f7f-w9972"]
Oct 30 14:07:27.884714 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.884609    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fda850-fa06-40c9-914e-0ad7ecde464c" path="/var/lib/kubelet/pods/52fda850-fa06-40c9-914e-0ad7ecde464c/volumes"
Oct 30 14:07:27.885728 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.885701    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="569d1268-67e4-4ff7-a710-9183c91fdc93" path="/var/lib/kubelet/pods/569d1268-67e4-4ff7-a710-9183c91fdc93/volumes"
Oct 30 14:07:27.886771 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:27.886748    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" path="/var/lib/kubelet/pods/67fb9d16-7c3e-4a74-93b6-0cede5462c07/volumes"
Oct 30 14:07:28.531038 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531003    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"]
Oct 30 14:07:28.531555 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531542    4291 cpu_manager.go:414] "containerMap: removing container" podUID="569d1268-67e4-4ff7-a710-9183c91fdc93" containerName="revoke-keys"
Oct 30 14:07:28.531602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531556    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="569d1268-67e4-4ff7-a710-9183c91fdc93" containerName="revoke-keys"
Oct 30 14:07:28.531602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531568    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ed7440e2-c465-496c-8172-161a37301ab9" containerName="payload-pre-processing"
Oct 30 14:07:28.531602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531574    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed7440e2-c465-496c-8172-161a37301ab9" containerName="payload-pre-processing"
Oct 30 14:07:28.531602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531593    4291 cpu_manager.go:414] "containerMap: removing container" podUID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" containerName="payload-processing"
Oct 30 14:07:28.531602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531597    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" containerName="payload-processing"
Oct 30 14:07:28.531803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531609    4291 cpu_manager.go:414] "containerMap: removing container" podUID="52fda850-fa06-40c9-914e-0ad7ecde464c" containerName="payload-pre-processing"
Oct 30 14:07:28.531803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531613    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fda850-fa06-40c9-914e-0ad7ecde464c" containerName="payload-pre-processing"
Oct 30 14:07:28.531803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531764    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fda850-fa06-40c9-914e-0ad7ecde464c" containerName="payload-pre-processing"
Oct 30 14:07:28.531803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531782    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="569d1268-67e4-4ff7-a710-9183c91fdc93" containerName="revoke-keys"
Oct 30 14:07:28.531803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531788    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed7440e2-c465-496c-8172-161a37301ab9" containerName="payload-pre-processing"
Oct 30 14:07:28.531803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.531797    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="67fb9d16-7c3e-4a74-93b6-0cede5462c07" containerName="payload-processing"
Oct 30 14:07:28.536698 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.536677    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.550778 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.550628    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:07:28.552200 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.552173    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"]
Oct 30 14:07:28.560563 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.560539    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:07:28.560739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.560583    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:07:28.560739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.560611    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:07:28.627601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.627568    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbbf8\" (UniqueName: \"kubernetes.io/projected/83866d27-a6c4-4198-98ef-40817bab0941-kube-api-access-bbbf8\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.628040 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.627700    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.628040 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.627750    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.729505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.729467    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.729714 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.729516    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.729767 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.729747    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbbf8\" (UniqueName: \"kubernetes.io/projected/83866d27-a6c4-4198-98ef-40817bab0941-kube-api-access-bbbf8\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.730204 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.730172    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.731999 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.731975    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.748431 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.748399    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbbf8\" (UniqueName: \"kubernetes.io/projected/83866d27-a6c4-4198-98ef-40817bab0941-kube-api-access-bbbf8\") pod \"maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.822705 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.822614    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"]
Oct 30 14:07:28.826898 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.826880    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:28.842950 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.842914    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"]
Oct 30 14:07:28.851046 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.851013    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:28.932743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.932349    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:28.932743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.932448    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b52lx\" (UniqueName: \"kubernetes.io/projected/b5606a9d-5078-4203-a318-0ae43169f13b-kube-api-access-b52lx\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:28.932743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:28.932505    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.034180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.034142    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b52lx\" (UniqueName: \"kubernetes.io/projected/b5606a9d-5078-4203-a318-0ae43169f13b-kube-api-access-b52lx\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.034362 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.034198    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.034437 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.034417    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.035110 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.035085    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.036663 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.036622    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.053088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.053064    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b52lx\" (UniqueName: \"kubernetes.io/projected/b5606a9d-5078-4203-a318-0ae43169f13b-kube-api-access-b52lx\") pod \"maas-api-revoke-sub-aed44bf0-5sknv\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.142129 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.142095    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:29.166701 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.166675    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"]
Oct 30 14:07:29.168777 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:07:29.168747    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83866d27_a6c4_4198_98ef_40817bab0941.slice/crio-5c6b6cb0c3365a9ea5094b3ba126abd8abe7f95f79b9c941dea540949fe3c96a WatchSource:0}: Error finding container 5c6b6cb0c3365a9ea5094b3ba126abd8abe7f95f79b9c941dea540949fe3c96a: Status 404 returned error can't find the container with id 5c6b6cb0c3365a9ea5094b3ba126abd8abe7f95f79b9c941dea540949fe3c96a
Oct 30 14:07:29.451938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.451907    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"]
Oct 30 14:07:29.453725 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:07:29.453694    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5606a9d_5078_4203_a318_0ae43169f13b.slice/crio-ffb0f799136232ff8cb074631087e31a43560173ad0d679c1bc6b1383467fa0e WatchSource:0}: Error finding container ffb0f799136232ff8cb074631087e31a43560173ad0d679c1bc6b1383467fa0e: Status 404 returned error can't find the container with id ffb0f799136232ff8cb074631087e31a43560173ad0d679c1bc6b1383467fa0e
Oct 30 14:07:29.623448 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.623406    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv" event={"ID":"b5606a9d-5078-4203-a318-0ae43169f13b","Type":"ContainerStarted","Data":"ffb0f799136232ff8cb074631087e31a43560173ad0d679c1bc6b1383467fa0e"}
Oct 30 14:07:29.625950 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.625921    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn" event={"ID":"83866d27-a6c4-4198-98ef-40817bab0941","Type":"ContainerStarted","Data":"ab9164de3711c46eb547baba7c72bd7a64f7e71f2f6fd7aa51408aa249a8f9f3"}
Oct 30 14:07:29.626099 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.625973    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn" event={"ID":"83866d27-a6c4-4198-98ef-40817bab0941","Type":"ContainerStarted","Data":"5c6b6cb0c3365a9ea5094b3ba126abd8abe7f95f79b9c941dea540949fe3c96a"}
Oct 30 14:07:29.675510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:29.675456    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn" podStartSLOduration=1.675438442 podStartE2EDuration="1.675438442s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:29.672781486 +0000 UTC m=+3859.933019720" watchObservedRunningTime="2026-10-30 14:07:29.675438442 +0000 UTC m=+3859.935676678"
Oct 30 14:07:30.634408 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:30.634365    4291 generic.go:356] "Generic (PLEG): container finished" podID="83866d27-a6c4-4198-98ef-40817bab0941" containerID="ab9164de3711c46eb547baba7c72bd7a64f7e71f2f6fd7aa51408aa249a8f9f3" exitCode=0
Oct 30 14:07:30.634602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:30.634404    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn" event={"ID":"83866d27-a6c4-4198-98ef-40817bab0941","Type":"ContainerDied","Data":"ab9164de3711c46eb547baba7c72bd7a64f7e71f2f6fd7aa51408aa249a8f9f3"}
Oct 30 14:07:30.636221 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:30.636198    4291 generic.go:356] "Generic (PLEG): container finished" podID="b5606a9d-5078-4203-a318-0ae43169f13b" containerID="b6b6490ee41dc3c4ca42356c47844ea5446ea3dfe0838083328f3c60950430a1" exitCode=0
Oct 30 14:07:30.636336 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:30.636256    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv" event={"ID":"b5606a9d-5078-4203-a318-0ae43169f13b","Type":"ContainerDied","Data":"b6b6490ee41dc3c4ca42356c47844ea5446ea3dfe0838083328f3c60950430a1"}
Oct 30 14:07:31.824452 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.824424    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:31.827922 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.827899    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:31.897526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.897451    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-xks-ca\") pod \"83866d27-a6c4-4198-98ef-40817bab0941\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") "
Oct 30 14:07:31.897735 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.897630    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-openshift-ca\") pod \"b5606a9d-5078-4203-a318-0ae43169f13b\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") "
Oct 30 14:07:31.897735 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.897682    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-openshift-ca\") pod \"83866d27-a6c4-4198-98ef-40817bab0941\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") "
Oct 30 14:07:31.897895 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.897837    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83866d27-a6c4-4198-98ef-40817bab0941-kube-api-access-bbbf8\" (UniqueName: \"kubernetes.io/projected/83866d27-a6c4-4198-98ef-40817bab0941-kube-api-access-bbbf8\") pod \"83866d27-a6c4-4198-98ef-40817bab0941\" (UID: \"83866d27-a6c4-4198-98ef-40817bab0941\") "
Oct 30 14:07:31.897895 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.897880    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-xks-ca\") pod \"b5606a9d-5078-4203-a318-0ae43169f13b\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") "
Oct 30 14:07:31.898014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.897914    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5606a9d-5078-4203-a318-0ae43169f13b-kube-api-access-b52lx\" (UniqueName: \"kubernetes.io/projected/b5606a9d-5078-4203-a318-0ae43169f13b-kube-api-access-b52lx\") pod \"b5606a9d-5078-4203-a318-0ae43169f13b\" (UID: \"b5606a9d-5078-4203-a318-0ae43169f13b\") "
Oct 30 14:07:31.898401 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.898299    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-openshift-ca" pod "b5606a9d-5078-4203-a318-0ae43169f13b" (UID: "b5606a9d-5078-4203-a318-0ae43169f13b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:31.898627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.898597    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:31.898933 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.898912    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-openshift-ca" pod "83866d27-a6c4-4198-98ef-40817bab0941" (UID: "83866d27-a6c4-4198-98ef-40817bab0941"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:31.903975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.902260    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5606a9d-5078-4203-a318-0ae43169f13b-kube-api-access-b52lx" pod "b5606a9d-5078-4203-a318-0ae43169f13b" (UID: "b5606a9d-5078-4203-a318-0ae43169f13b"). InnerVolumeSpecName "kube-api-access-b52lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:31.903975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.902351    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-xks-ca" pod "b5606a9d-5078-4203-a318-0ae43169f13b" (UID: "b5606a9d-5078-4203-a318-0ae43169f13b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:31.907884 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.904482    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83866d27-a6c4-4198-98ef-40817bab0941-kube-api-access-bbbf8" pod "83866d27-a6c4-4198-98ef-40817bab0941" (UID: "83866d27-a6c4-4198-98ef-40817bab0941"). InnerVolumeSpecName "kube-api-access-bbbf8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:31.908315 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:31.908219    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-xks-ca" pod "83866d27-a6c4-4198-98ef-40817bab0941" (UID: "83866d27-a6c4-4198-98ef-40817bab0941"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:32.000185 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.000150    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:32.000185 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.000184    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbbf8\" (UniqueName: \"kubernetes.io/projected/83866d27-a6c4-4198-98ef-40817bab0941-kube-api-access-bbbf8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:32.000374 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.000199    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5606a9d-5078-4203-a318-0ae43169f13b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:32.000374 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.000211    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b52lx\" (UniqueName: \"kubernetes.io/projected/b5606a9d-5078-4203-a318-0ae43169f13b-kube-api-access-b52lx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:32.000374 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.000223    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83866d27-a6c4-4198-98ef-40817bab0941-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:32.651127 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.651094    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"
Oct 30 14:07:32.651127 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.651103    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv" event={"ID":"b5606a9d-5078-4203-a318-0ae43169f13b","Type":"ContainerDied","Data":"ffb0f799136232ff8cb074631087e31a43560173ad0d679c1bc6b1383467fa0e"}
Oct 30 14:07:32.651522 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.651156    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffb0f799136232ff8cb074631087e31a43560173ad0d679c1bc6b1383467fa0e"
Oct 30 14:07:32.652591 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.652570    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"
Oct 30 14:07:32.652748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.652609    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn" event={"ID":"83866d27-a6c4-4198-98ef-40817bab0941","Type":"ContainerDied","Data":"5c6b6cb0c3365a9ea5094b3ba126abd8abe7f95f79b9c941dea540949fe3c96a"}
Oct 30 14:07:32.652748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:32.652665    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6b6cb0c3365a9ea5094b3ba126abd8abe7f95f79b9c941dea540949fe3c96a"
Oct 30 14:07:33.268763 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:33.268731    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"]
Oct 30 14:07:33.273874 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:33.273848    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"]
Oct 30 14:07:33.289398 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:33.289374    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"]
Oct 30 14:07:36.497256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:36.497213    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"]
Oct 30 14:07:36.497867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:36.497477    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy" containerID="cri-o://d069b140075da6c91605cec6fb8fcae2df6fce6093314a37f4b6ee5fe9dddfef" gracePeriod=30
Oct 30 14:07:37.102308 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.102268    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"]
Oct 30 14:07:37.102916 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.102900    4291 cpu_manager.go:414] "containerMap: removing container" podUID="83866d27-a6c4-4198-98ef-40817bab0941" containerName="revoke-keys"
Oct 30 14:07:37.102972 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.102918    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83866d27-a6c4-4198-98ef-40817bab0941" containerName="revoke-keys"
Oct 30 14:07:37.102972 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.102936    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b5606a9d-5078-4203-a318-0ae43169f13b" containerName="revoke-keys"
Oct 30 14:07:37.102972 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.102942    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5606a9d-5078-4203-a318-0ae43169f13b" containerName="revoke-keys"
Oct 30 14:07:37.103093 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.103037    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="83866d27-a6c4-4198-98ef-40817bab0941" containerName="revoke-keys"
Oct 30 14:07:37.103093 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.103047    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5606a9d-5078-4203-a318-0ae43169f13b" containerName="revoke-keys"
Oct 30 14:07:37.107035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.107015    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.126838 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.126802    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"]
Oct 30 14:07:37.133567 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.133534    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:07:37.133567 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.133534    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:07:37.189152 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.189117    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.189340 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.189183    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.189389 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.189352    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725hf\" (UniqueName: \"kubernetes.io/projected/343d1d3c-16c2-4a08-bf85-b27ef588c52b-kube-api-access-725hf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.191392 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.191369    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:07:37.191510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.191476    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:07:37.290550 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.290504    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725hf\" (UniqueName: \"kubernetes.io/projected/343d1d3c-16c2-4a08-bf85-b27ef588c52b-kube-api-access-725hf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.290807 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.290678    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.290807 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.290735    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.291438 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.291409    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.293052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.293033    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.310306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.310276    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725hf\" (UniqueName: \"kubernetes.io/projected/343d1d3c-16c2-4a08-bf85-b27ef588c52b-kube-api-access-725hf\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.421581 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.421484    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:37.730993 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:37.730958    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"]
Oct 30 14:07:37.732306 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:07:37.732280    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343d1d3c_16c2_4a08_bf85_b27ef588c52b.slice/crio-273f83b72d94702010085b189c64e6207625c300bfcba5baa7dfabaa60ead258 WatchSource:0}: Error finding container 273f83b72d94702010085b189c64e6207625c300bfcba5baa7dfabaa60ead258: Status 404 returned error can't find the container with id 273f83b72d94702010085b189c64e6207625c300bfcba5baa7dfabaa60ead258
Oct 30 14:07:38.211463 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:07:38.211420    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:07:38.719986 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:38.719944    4291 generic.go:356] "Generic (PLEG): container finished" podID="343d1d3c-16c2-4a08-bf85-b27ef588c52b" containerID="03150863e31d0cdc861836a31837608997e187f5b65acb27cb1e0f7b44844b0d" exitCode=0
Oct 30 14:07:38.720168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:38.720025    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm" event={"ID":"343d1d3c-16c2-4a08-bf85-b27ef588c52b","Type":"ContainerDied","Data":"03150863e31d0cdc861836a31837608997e187f5b65acb27cb1e0f7b44844b0d"}
Oct 30 14:07:38.720168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:38.720088    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm" event={"ID":"343d1d3c-16c2-4a08-bf85-b27ef588c52b","Type":"ContainerStarted","Data":"273f83b72d94702010085b189c64e6207625c300bfcba5baa7dfabaa60ead258"}
Oct 30 14:07:39.197808 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.189613    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"]
Oct 30 14:07:39.202489 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.202458    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.218028 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.217994    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"]
Oct 30 14:07:39.219002 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.218962    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4xkt\" (UniqueName: \"kubernetes.io/projected/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-kube-api-access-m4xkt\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.219233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.219215    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.219610 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.219592    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.320990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.320946    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.321203 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.321054    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4xkt\" (UniqueName: \"kubernetes.io/projected/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-kube-api-access-m4xkt\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.321203 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.321123    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.321803 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.321775    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.323728 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.323707    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.340184 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.340151    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4xkt\" (UniqueName: \"kubernetes.io/projected/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-kube-api-access-m4xkt\") pod \"maas-api-revoke-sub-17297c37-rsdsg\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.536712 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.535883    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:39.892892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.892861    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:39.936334 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.936303    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/343d1d3c-16c2-4a08-bf85-b27ef588c52b-kube-api-access-725hf\" (UniqueName: \"kubernetes.io/projected/343d1d3c-16c2-4a08-bf85-b27ef588c52b-kube-api-access-725hf\") pod \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") "
Oct 30 14:07:39.936520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.936379    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-xks-ca\") pod \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") "
Oct 30 14:07:39.936649 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.936549    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-openshift-ca\") pod \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\" (UID: \"343d1d3c-16c2-4a08-bf85-b27ef588c52b\") "
Oct 30 14:07:39.937347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.936958    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-openshift-ca" pod "343d1d3c-16c2-4a08-bf85-b27ef588c52b" (UID: "343d1d3c-16c2-4a08-bf85-b27ef588c52b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:39.937734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.937547    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:39.938739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.938709    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-xks-ca" pod "343d1d3c-16c2-4a08-bf85-b27ef588c52b" (UID: "343d1d3c-16c2-4a08-bf85-b27ef588c52b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:39.939003 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.938979    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/343d1d3c-16c2-4a08-bf85-b27ef588c52b-kube-api-access-725hf" pod "343d1d3c-16c2-4a08-bf85-b27ef588c52b" (UID: "343d1d3c-16c2-4a08-bf85-b27ef588c52b"). InnerVolumeSpecName "kube-api-access-725hf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:39.940038 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:07:39.940006    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10dbb17d_fc1b_4d4f_9908_62cb19a7fa41.slice/crio-86c424aa87a0c1118f982affb97037ec266a0151ddfd058686d5298bc9bfed73 WatchSource:0}: Error finding container 86c424aa87a0c1118f982affb97037ec266a0151ddfd058686d5298bc9bfed73: Status 404 returned error can't find the container with id 86c424aa87a0c1118f982affb97037ec266a0151ddfd058686d5298bc9bfed73
Oct 30 14:07:39.953693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:39.947310    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"]
Oct 30 14:07:40.038792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.038751    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725hf\" (UniqueName: \"kubernetes.io/projected/343d1d3c-16c2-4a08-bf85-b27ef588c52b-kube-api-access-725hf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:40.038792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.038783    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/343d1d3c-16c2-4a08-bf85-b27ef588c52b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:40.737419 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.737383    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"
Oct 30 14:07:40.737419 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.737396    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm" event={"ID":"343d1d3c-16c2-4a08-bf85-b27ef588c52b","Type":"ContainerDied","Data":"273f83b72d94702010085b189c64e6207625c300bfcba5baa7dfabaa60ead258"}
Oct 30 14:07:40.738083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.737447    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="273f83b72d94702010085b189c64e6207625c300bfcba5baa7dfabaa60ead258"
Oct 30 14:07:40.739467 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.739437    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg" event={"ID":"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41","Type":"ContainerStarted","Data":"f9921eba83d10af26318c9fa68c20d9e9b15559e429b53c48912f009f5344dbc"}
Oct 30 14:07:40.739618 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.739489    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg" event={"ID":"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41","Type":"ContainerStarted","Data":"86c424aa87a0c1118f982affb97037ec266a0151ddfd058686d5298bc9bfed73"}
Oct 30 14:07:40.778782 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:40.778734    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg" podStartSLOduration=1.7787208140000001 podStartE2EDuration="1.778720814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:07:40.776321908 +0000 UTC m=+3871.036560144" watchObservedRunningTime="2026-10-30 14:07:40.778720814 +0000 UTC m=+3871.038959082"
Oct 30 14:07:41.750316 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.749977    4291 generic.go:356] "Generic (PLEG): container finished" podID="10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" containerID="f9921eba83d10af26318c9fa68c20d9e9b15559e429b53c48912f009f5344dbc" exitCode=0
Oct 30 14:07:41.750316 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.750248    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg" event={"ID":"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41","Type":"ContainerDied","Data":"f9921eba83d10af26318c9fa68c20d9e9b15559e429b53c48912f009f5344dbc"}
Oct 30 14:07:41.753575 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.753537    4291 generic.go:356] "Generic (PLEG): container finished" podID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerID="d069b140075da6c91605cec6fb8fcae2df6fce6093314a37f4b6ee5fe9dddfef" exitCode=0
Oct 30 14:07:41.753760 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.753603    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" event={"ID":"d1a4a2eb-20d2-48bf-afc8-cb6900350068","Type":"ContainerDied","Data":"d069b140075da6c91605cec6fb8fcae2df6fce6093314a37f4b6ee5fe9dddfef"}
Oct 30 14:07:41.753760 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.753689    4291 scope.go:122] "RemoveContainer" containerID="1d43bfc97521fa6de08dbeeaf2bff1b1543abaaa3d3dee84a8f626bc97bcfd7d"
Oct 30 14:07:41.807898 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.807868    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 14:07:41.867106 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867070    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-podinfo\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867106 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867113    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istiod-ca-cert\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867388 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867142    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-certs\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867388 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867161    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-socket\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867388 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867271    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-kube-api-access-8qt2l\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-kube-api-access-8qt2l\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867388 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867287    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-token\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-token\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867388 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867307    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-envoy\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867388 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867327    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-credential-socket\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867755 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867492    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-data\") pod \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\" (UID: \"d1a4a2eb-20d2-48bf-afc8-cb6900350068\") "
Oct 30 14:07:41.867755 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867697    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-credential-socket" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:41.868017 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.867994    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istiod-ca-cert" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:41.868239 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.868192    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-data" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:41.868354 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.868336    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.868466 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.868360    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.868466 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.868375    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.869894 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.869866    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-podinfo" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:07:41.870102 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.870085    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-certs" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:41.871907 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.871880    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-envoy" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:41.872065 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.871996    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-kube-api-access-8qt2l" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "kube-api-access-8qt2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:41.872152 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.872070    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-socket" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:41.872152 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.872115    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-token" pod "d1a4a2eb-20d2-48bf-afc8-cb6900350068" (UID: "d1a4a2eb-20d2-48bf-afc8-cb6900350068"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:41.969228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.969190    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.969228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.969228    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.969512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.969242    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.969512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.969255    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qt2l\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-kube-api-access-8qt2l\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.969512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.969264    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:41.969512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:41.969275    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1a4a2eb-20d2-48bf-afc8-cb6900350068-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:42.763380 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.763338    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"
Oct 30 14:07:42.763380 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.763349    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz" event={"ID":"d1a4a2eb-20d2-48bf-afc8-cb6900350068","Type":"ContainerDied","Data":"c3b3d9b92081ba88720c4ea573107136316d22ff539c4754a53c633c8ebceb04"}
Oct 30 14:07:42.764062 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.763424    4291 scope.go:122] "RemoveContainer" containerID="d069b140075da6c91605cec6fb8fcae2df6fce6093314a37f4b6ee5fe9dddfef"
Oct 30 14:07:42.828041 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.828014    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"]
Oct 30 14:07:42.836682 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.832789    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-d2cecb-openshift-default-69d6b6fdb5-fmcvz"]
Oct 30 14:07:42.939523 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.939493    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:42.982263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.981904    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-kube-api-access-m4xkt\" (UniqueName: \"kubernetes.io/projected/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-kube-api-access-m4xkt\") pod \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") "
Oct 30 14:07:42.982263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.982048    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-openshift-ca\") pod \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") "
Oct 30 14:07:42.982263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.982087    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-xks-ca\") pod \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\" (UID: \"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41\") "
Oct 30 14:07:42.983679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.983627    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-openshift-ca" pod "10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" (UID: "10dbb17d-fc1b-4d4f-9908-62cb19a7fa41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:42.984342 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.984307    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:42.987778 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.987752    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-xks-ca" pod "10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" (UID: "10dbb17d-fc1b-4d4f-9908-62cb19a7fa41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:42.987937 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:42.987834    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-kube-api-access-m4xkt" pod "10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" (UID: "10dbb17d-fc1b-4d4f-9908-62cb19a7fa41"). InnerVolumeSpecName "kube-api-access-m4xkt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:43.086275 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:43.086194    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:43.086275 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:43.086223    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4xkt\" (UniqueName: \"kubernetes.io/projected/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41-kube-api-access-m4xkt\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:43.772800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:43.772761    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"
Oct 30 14:07:43.772800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:43.772797    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg" event={"ID":"10dbb17d-fc1b-4d4f-9908-62cb19a7fa41","Type":"ContainerDied","Data":"86c424aa87a0c1118f982affb97037ec266a0151ddfd058686d5298bc9bfed73"}
Oct 30 14:07:43.773281 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:43.772863    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86c424aa87a0c1118f982affb97037ec266a0151ddfd058686d5298bc9bfed73"
Oct 30 14:07:43.880403 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:43.880365    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" path="/var/lib/kubelet/pods/d1a4a2eb-20d2-48bf-afc8-cb6900350068/volumes"
Oct 30 14:07:46.038647 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:46.038607    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"]
Oct 30 14:07:46.063660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:46.060282    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f8858b0-6z4vl"]
Oct 30 14:07:47.879302 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:47.879265    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40faca54-8328-4608-9d45-5b13343021fd" path="/var/lib/kubelet/pods/40faca54-8328-4608-9d45-5b13343021fd/volumes"
Oct 30 14:07:48.290559 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.290470    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main" containerID="cri-o://e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9" gracePeriod=45
Oct 30 14:07:48.294685 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.294620    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main" containerID="cri-o://39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a" gracePeriod=45
Oct 30 14:07:48.309688 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.309612    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" containerID="cri-o://154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02" gracePeriod=45
Oct 30 14:07:48.339836 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.339798    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.39:8000/ready\": dial tcp 10.133.0.39:8000: connect: connection refused"
Oct 30 14:07:48.599088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.599064    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 14:07:48.610025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.609991    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 14:07:48.612081 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.612061    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 14:07:48.675703 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.675649    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-home\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-home\") pod \"d5779532-3650-4896-95ee-35329a605600\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.675731    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46c01365-1530-4b25-829b-072cdbadb706-kube-api-access-2tkh4\" (UniqueName: \"kubernetes.io/projected/46c01365-1530-4b25-829b-072cdbadb706-kube-api-access-2tkh4\") pod \"46c01365-1530-4b25-829b-072cdbadb706\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.675773    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tls-certs\") pod \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.675805    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-dshm\") pod \"d5779532-3650-4896-95ee-35329a605600\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.675857    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-tmp-dir\") pod \"d5779532-3650-4896-95ee-35329a605600\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.675963    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-dshm\") pod \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.675993    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-home\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-home\") pod \"46c01365-1530-4b25-829b-072cdbadb706\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676030    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5779532-3650-4896-95ee-35329a605600-tls-certs\" (UniqueName: \"kubernetes.io/secret/d5779532-3650-4896-95ee-35329a605600-tls-certs\") pod \"d5779532-3650-4896-95ee-35329a605600\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676036    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-home" pod "d5779532-3650-4896-95ee-35329a605600" (UID: "d5779532-3650-4896-95ee-35329a605600"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676118    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-model-cache\") pod \"46c01365-1530-4b25-829b-072cdbadb706\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676316    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5779532-3650-4896-95ee-35329a605600-kube-api-access-5tm2f\" (UniqueName: \"kubernetes.io/projected/d5779532-3650-4896-95ee-35329a605600-kube-api-access-5tm2f\") pod \"d5779532-3650-4896-95ee-35329a605600\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676352    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-model-cache\") pod \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676397    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-model-cache\") pod \"d5779532-3650-4896-95ee-35329a605600\" (UID: \"d5779532-3650-4896-95ee-35329a605600\") "
Oct 30 14:07:48.676456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676452    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46c01365-1530-4b25-829b-072cdbadb706-tls-certs\" (UniqueName: \"kubernetes.io/secret/46c01365-1530-4b25-829b-072cdbadb706-tls-certs\") pod \"46c01365-1530-4b25-829b-072cdbadb706\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") "
Oct 30 14:07:48.677275 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676493    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-tmp-dir\") pod \"46c01365-1530-4b25-829b-072cdbadb706\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") "
Oct 30 14:07:48.677275 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676518    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-dshm\") pod \"46c01365-1530-4b25-829b-072cdbadb706\" (UID: \"46c01365-1530-4b25-829b-072cdbadb706\") "
Oct 30 14:07:48.677275 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676564    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-home\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-home\") pod \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") "
Oct 30 14:07:48.677275 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676605    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tmp-dir\") pod \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") "
Oct 30 14:07:48.677275 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.676663    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc32b36-6fdb-4bfc-8d55-6240290271b9-kube-api-access-l8d56\" (UniqueName: \"kubernetes.io/projected/7fc32b36-6fdb-4bfc-8d55-6240290271b9-kube-api-access-l8d56\") pod \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\" (UID: \"7fc32b36-6fdb-4bfc-8d55-6240290271b9\") "
Oct 30 14:07:48.677611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.677363    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-tmp-dir" pod "d5779532-3650-4896-95ee-35329a605600" (UID: "d5779532-3650-4896-95ee-35329a605600"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.680499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.679301    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-model-cache" pod "d5779532-3650-4896-95ee-35329a605600" (UID: "d5779532-3650-4896-95ee-35329a605600"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.680499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.679647    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-model-cache" pod "46c01365-1530-4b25-829b-072cdbadb706" (UID: "46c01365-1530-4b25-829b-072cdbadb706"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.680499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.679864    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-home" pod "7fc32b36-6fdb-4bfc-8d55-6240290271b9" (UID: "7fc32b36-6fdb-4bfc-8d55-6240290271b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.680499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.680223    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-tmp-dir" pod "46c01365-1530-4b25-829b-072cdbadb706" (UID: "46c01365-1530-4b25-829b-072cdbadb706"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.680499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.680269    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-home" pod "46c01365-1530-4b25-829b-072cdbadb706" (UID: "46c01365-1530-4b25-829b-072cdbadb706"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.680499 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.680442    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tmp-dir" pod "7fc32b36-6fdb-4bfc-8d55-6240290271b9" (UID: "7fc32b36-6fdb-4bfc-8d55-6240290271b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.681521 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.681481    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-model-cache" pod "7fc32b36-6fdb-4bfc-8d55-6240290271b9" (UID: "7fc32b36-6fdb-4bfc-8d55-6240290271b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.682567 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682537    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tls-certs" pod "7fc32b36-6fdb-4bfc-8d55-6240290271b9" (UID: "7fc32b36-6fdb-4bfc-8d55-6240290271b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:48.682825 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682782    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682825 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682814    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682831    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682845    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682858    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682871    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682883    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682896    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682908    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.682922    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.688738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.688684    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc32b36-6fdb-4bfc-8d55-6240290271b9-kube-api-access-l8d56" pod "7fc32b36-6fdb-4bfc-8d55-6240290271b9" (UID: "7fc32b36-6fdb-4bfc-8d55-6240290271b9"). InnerVolumeSpecName "kube-api-access-l8d56". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:48.692251 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.692210    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46c01365-1530-4b25-829b-072cdbadb706-kube-api-access-2tkh4" pod "46c01365-1530-4b25-829b-072cdbadb706" (UID: "46c01365-1530-4b25-829b-072cdbadb706"). InnerVolumeSpecName "kube-api-access-2tkh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:48.693249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.692368    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-dshm" pod "d5779532-3650-4896-95ee-35329a605600" (UID: "d5779532-3650-4896-95ee-35329a605600"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.693249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.692992    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46c01365-1530-4b25-829b-072cdbadb706-tls-certs" pod "46c01365-1530-4b25-829b-072cdbadb706" (UID: "46c01365-1530-4b25-829b-072cdbadb706"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:48.693249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.693100    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5779532-3650-4896-95ee-35329a605600-tls-certs" pod "d5779532-3650-4896-95ee-35329a605600" (UID: "d5779532-3650-4896-95ee-35329a605600"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:48.693249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.693159    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-dshm" pod "7fc32b36-6fdb-4bfc-8d55-6240290271b9" (UID: "7fc32b36-6fdb-4bfc-8d55-6240290271b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.693932 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.693911    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-dshm" pod "46c01365-1530-4b25-829b-072cdbadb706" (UID: "46c01365-1530-4b25-829b-072cdbadb706"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:07:48.694369 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.694341    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5779532-3650-4896-95ee-35329a605600-kube-api-access-5tm2f" pod "d5779532-3650-4896-95ee-35329a605600" (UID: "d5779532-3650-4896-95ee-35329a605600"). InnerVolumeSpecName "kube-api-access-5tm2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:48.784098 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784060    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46c01365-1530-4b25-829b-072cdbadb706-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.784098 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784096    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46c01365-1530-4b25-829b-072cdbadb706-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.784319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784110    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8d56\" (UniqueName: \"kubernetes.io/projected/7fc32b36-6fdb-4bfc-8d55-6240290271b9-kube-api-access-l8d56\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.784319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784124    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tkh4\" (UniqueName: \"kubernetes.io/projected/46c01365-1530-4b25-829b-072cdbadb706-kube-api-access-2tkh4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.784319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784138    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5779532-3650-4896-95ee-35329a605600-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.784319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784148    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc32b36-6fdb-4bfc-8d55-6240290271b9-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.784319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784159    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5779532-3650-4896-95ee-35329a605600-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.784319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.784171    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tm2f\" (UniqueName: \"kubernetes.io/projected/d5779532-3650-4896-95ee-35329a605600-kube-api-access-5tm2f\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:48.815118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.815029    4291 generic.go:356] "Generic (PLEG): container finished" podID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerID="154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02" exitCode=0
Oct 30 14:07:48.815118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.815077    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"
Oct 30 14:07:48.815118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.815112    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" event={"ID":"7fc32b36-6fdb-4bfc-8d55-6240290271b9","Type":"ContainerDied","Data":"154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02"}
Oct 30 14:07:48.815362 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.815155    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw" event={"ID":"7fc32b36-6fdb-4bfc-8d55-6240290271b9","Type":"ContainerDied","Data":"357741f6396a07c1f768dfa3118da7b035c7cba898dbd9332ab26b40606b9574"}
Oct 30 14:07:48.815362 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.815182    4291 scope.go:122] "RemoveContainer" containerID="154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02"
Oct 30 14:07:48.816611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.816592    4291 generic.go:356] "Generic (PLEG): container finished" podID="d5779532-3650-4896-95ee-35329a605600" containerID="e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9" exitCode=0
Oct 30 14:07:48.816756 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.816655    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"
Oct 30 14:07:48.816756 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.816682    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" event={"ID":"d5779532-3650-4896-95ee-35329a605600","Type":"ContainerDied","Data":"e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9"}
Oct 30 14:07:48.816756 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.816716    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq" event={"ID":"d5779532-3650-4896-95ee-35329a605600","Type":"ContainerDied","Data":"fdd4335a5dca9b55a8e909831c9ca213ce1702335746fb6579f1a7697be81358"}
Oct 30 14:07:48.818179 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.818155    4291 generic.go:356] "Generic (PLEG): container finished" podID="46c01365-1530-4b25-829b-072cdbadb706" containerID="39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a" exitCode=0
Oct 30 14:07:48.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.818194    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" event={"ID":"46c01365-1530-4b25-829b-072cdbadb706","Type":"ContainerDied","Data":"39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a"}
Oct 30 14:07:48.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.818201    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"
Oct 30 14:07:48.818306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.818214    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p" event={"ID":"46c01365-1530-4b25-829b-072cdbadb706","Type":"ContainerDied","Data":"3c0b70635f2c8a689159ccad05798c1af087560c5fd2975f0eddfff86119baf0"}
Oct 30 14:07:48.826453 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.826429    4291 scope.go:122] "RemoveContainer" containerID="154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02"
Oct 30 14:07:48.826769 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:07:48.826746    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02\": container with ID starting with 154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02 not found: ID does not exist" containerID="154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02"
Oct 30 14:07:48.826872 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.826775    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02"} err="failed to get container status \"154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02\": rpc error: code = NotFound desc = could not find container \"154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02\": container with ID starting with 154dd0ae37d2a0476754e8393b22759af60a03d09ba8d73e2fd993521b777d02 not found: ID does not exist"
Oct 30 14:07:48.826872 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.826792    4291 scope.go:122] "RemoveContainer" containerID="e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9"
Oct 30 14:07:48.844571 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.844548    4291 scope.go:122] "RemoveContainer" containerID="e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9"
Oct 30 14:07:48.844931 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:07:48.844910    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9\": container with ID starting with e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9 not found: ID does not exist" containerID="e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9"
Oct 30 14:07:48.844977 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.844940    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9"} err="failed to get container status \"e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9\": rpc error: code = NotFound desc = could not find container \"e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9\": container with ID starting with e44075c255bf159f444048f9f9a91b062eac2160653f89f96b2824222e9a0cc9 not found: ID does not exist"
Oct 30 14:07:48.844977 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.844957    4291 scope.go:122] "RemoveContainer" containerID="39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a"
Oct 30 14:07:48.854386 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.854367    4291 scope.go:122] "RemoveContainer" containerID="39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a"
Oct 30 14:07:48.854677 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:07:48.854658    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a\": container with ID starting with 39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a not found: ID does not exist" containerID="39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a"
Oct 30 14:07:48.854753 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.854684    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a"} err="failed to get container status \"39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a\": rpc error: code = NotFound desc = could not find container \"39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a\": container with ID starting with 39667c9d993f9c351069dab8f8988a6e81bc4a535a7e22533d7503acddcfec5a not found: ID does not exist"
Oct 30 14:07:48.880052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.880017    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"]
Oct 30 14:07:48.883281 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.883255    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/simulator-w3-5852ff-kserve-54bb5ddbb8-pktjw"]
Oct 30 14:07:48.915137 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.915101    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"]
Oct 30 14:07:48.918088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.918065    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-w3-5852ff-kserve-678897f6b4-lc57p"]
Oct 30 14:07:48.948717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.948692    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"]
Oct 30 14:07:48.952442 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:48.952413    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5852ff/distinct-2-w3-5852ff-kserve-7b45fb556c-njnzq"]
Oct 30 14:07:49.878808 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:49.878771    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46c01365-1530-4b25-829b-072cdbadb706" path="/var/lib/kubelet/pods/46c01365-1530-4b25-829b-072cdbadb706/volumes"
Oct 30 14:07:49.879324 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:49.879308    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" path="/var/lib/kubelet/pods/7fc32b36-6fdb-4bfc-8d55-6240290271b9/volumes"
Oct 30 14:07:49.879752 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:49.879737    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5779532-3650-4896-95ee-35329a605600" path="/var/lib/kubelet/pods/d5779532-3650-4896-95ee-35329a605600/volumes"
Oct 30 14:07:50.010287 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010255    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"]
Oct 30 14:07:50.010831 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010816    4291 cpu_manager.go:414] "containerMap: removing container" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010833    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010855    4291 cpu_manager.go:414] "containerMap: removing container" podUID="343d1d3c-16c2-4a08-bf85-b27ef588c52b" containerName="revoke-keys"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010861    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="343d1d3c-16c2-4a08-bf85-b27ef588c52b" containerName="revoke-keys"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010872    4291 cpu_manager.go:414] "containerMap: removing container" podUID="10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" containerName="revoke-keys"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010877    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" containerName="revoke-keys"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010885    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010890    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010910    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010915    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010927    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010931    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010940    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy"
Oct 30 14:07:50.010941 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.010945    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy"
Oct 30 14:07:50.011341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.011018    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="343d1d3c-16c2-4a08-bf85-b27ef588c52b" containerName="revoke-keys"
Oct 30 14:07:50.011341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.011029    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" containerName="revoke-keys"
Oct 30 14:07:50.011341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.011037    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc32b36-6fdb-4bfc-8d55-6240290271b9" containerName="main"
Oct 30 14:07:50.011341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.011045    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5779532-3650-4896-95ee-35329a605600" containerName="main"
Oct 30 14:07:50.011341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.011051    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy"
Oct 30 14:07:50.011341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.011057    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1a4a2eb-20d2-48bf-afc8-cb6900350068" containerName="istio-proxy"
Oct 30 14:07:50.011341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.011062    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="46c01365-1530-4b25-829b-072cdbadb706" containerName="main"
Oct 30 14:07:50.015707 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.015688    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.030400 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.030372    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"]
Oct 30 14:07:50.038228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.038197    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:07:50.038426 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.038406    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:07:50.038426 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.038426    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:07:50.041417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.041396    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:07:50.106247 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.106207    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.106404 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.106254    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.106520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.106498    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whw4q\" (UniqueName: \"kubernetes.io/projected/bf8c180e-362e-483b-9fca-edcf1c593ea5-kube-api-access-whw4q\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.207699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.207579    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whw4q\" (UniqueName: \"kubernetes.io/projected/bf8c180e-362e-483b-9fca-edcf1c593ea5-kube-api-access-whw4q\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.207699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.207686    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.207963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.207737    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.208515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.208495    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.210193 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.210169    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.226279 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.226252    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whw4q\" (UniqueName: \"kubernetes.io/projected/bf8c180e-362e-483b-9fca-edcf1c593ea5-kube-api-access-whw4q\") pod \"maas-api-revoke-sub-33f32700-t9nlf\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.330777 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.330744    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:50.636951 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.636919    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"]
Oct 30 14:07:50.638229 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:07:50.638199    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf8c180e_362e_483b_9fca_edcf1c593ea5.slice/crio-05252ad50ecf26a8464b06a52da92b94b46fcaee328a5d33e5af7254f9955310 WatchSource:0}: Error finding container 05252ad50ecf26a8464b06a52da92b94b46fcaee328a5d33e5af7254f9955310: Status 404 returned error can't find the container with id 05252ad50ecf26a8464b06a52da92b94b46fcaee328a5d33e5af7254f9955310
Oct 30 14:07:50.836130 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:50.836088    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf" event={"ID":"bf8c180e-362e-483b-9fca-edcf1c593ea5","Type":"ContainerStarted","Data":"05252ad50ecf26a8464b06a52da92b94b46fcaee328a5d33e5af7254f9955310"}
Oct 30 14:07:51.844434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:51.844395    4291 generic.go:356] "Generic (PLEG): container finished" podID="bf8c180e-362e-483b-9fca-edcf1c593ea5" containerID="bb0a14da76166fa9754c8bf5dcc30657c2f420f234f8c8acc68dd0626ce8c065" exitCode=0
Oct 30 14:07:51.844835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:51.844488    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf" event={"ID":"bf8c180e-362e-483b-9fca-edcf1c593ea5","Type":"ContainerDied","Data":"bb0a14da76166fa9754c8bf5dcc30657c2f420f234f8c8acc68dd0626ce8c065"}
Oct 30 14:07:53.030426 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.030392    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:53.152236 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.152097    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-xks-ca\") pod \"bf8c180e-362e-483b-9fca-edcf1c593ea5\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") "
Oct 30 14:07:53.152507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.152483    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-openshift-ca\") pod \"bf8c180e-362e-483b-9fca-edcf1c593ea5\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") "
Oct 30 14:07:53.152591 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.152554    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf8c180e-362e-483b-9fca-edcf1c593ea5-kube-api-access-whw4q\" (UniqueName: \"kubernetes.io/projected/bf8c180e-362e-483b-9fca-edcf1c593ea5-kube-api-access-whw4q\") pod \"bf8c180e-362e-483b-9fca-edcf1c593ea5\" (UID: \"bf8c180e-362e-483b-9fca-edcf1c593ea5\") "
Oct 30 14:07:53.152978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.152949    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-openshift-ca" pod "bf8c180e-362e-483b-9fca-edcf1c593ea5" (UID: "bf8c180e-362e-483b-9fca-edcf1c593ea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:07:53.153240 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.153224    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:53.154825 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.154769    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-xks-ca" pod "bf8c180e-362e-483b-9fca-edcf1c593ea5" (UID: "bf8c180e-362e-483b-9fca-edcf1c593ea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:07:53.155074 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.155053    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf8c180e-362e-483b-9fca-edcf1c593ea5-kube-api-access-whw4q" pod "bf8c180e-362e-483b-9fca-edcf1c593ea5" (UID: "bf8c180e-362e-483b-9fca-edcf1c593ea5"). InnerVolumeSpecName "kube-api-access-whw4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:07:53.254537 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.254500    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whw4q\" (UniqueName: \"kubernetes.io/projected/bf8c180e-362e-483b-9fca-edcf1c593ea5-kube-api-access-whw4q\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:53.254537 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.254536    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf8c180e-362e-483b-9fca-edcf1c593ea5-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:07:53.861694 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.861619    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"
Oct 30 14:07:53.861694 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.861669    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf" event={"ID":"bf8c180e-362e-483b-9fca-edcf1c593ea5","Type":"ContainerDied","Data":"05252ad50ecf26a8464b06a52da92b94b46fcaee328a5d33e5af7254f9955310"}
Oct 30 14:07:53.862034 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:53.861728    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05252ad50ecf26a8464b06a52da92b94b46fcaee328a5d33e5af7254f9955310"
Oct 30 14:07:57.046912 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:57.046876    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"]
Oct 30 14:07:57.058468 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:57.058440    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ce33b1ad-fnn8d"]
Oct 30 14:07:57.879892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:57.879855    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78b116b8-d774-4c7b-a178-0e67c91904f4" path="/var/lib/kubelet/pods/78b116b8-d774-4c7b-a178-0e67c91904f4/volumes"
Oct 30 14:07:58.033249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:58.033206    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"]
Oct 30 14:07:58.044846 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:58.044814    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2773d612-n7hsf"]
Oct 30 14:07:59.879055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:07:59.879014    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5be8050e-8c6d-4e21-831d-602335f316c6" path="/var/lib/kubelet/pods/5be8050e-8c6d-4e21-831d-602335f316c6/volumes"
Oct 30 14:08:00.246164 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.246067    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"]
Oct 30 14:08:00.246630 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.246616    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bf8c180e-362e-483b-9fca-edcf1c593ea5" containerName="revoke-keys"
Oct 30 14:08:00.246708 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.246645    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf8c180e-362e-483b-9fca-edcf1c593ea5" containerName="revoke-keys"
Oct 30 14:08:00.246806 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.246798    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf8c180e-362e-483b-9fca-edcf1c593ea5" containerName="revoke-keys"
Oct 30 14:08:00.251427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.251404    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.265446 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.265422    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"]
Oct 30 14:08:00.269608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.269582    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.272482 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.272454    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"]
Oct 30 14:08:00.274365 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.274334    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:08:00.274502 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.274371    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:08:00.274502 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.274379    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:08:00.280685 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.280626    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:08:00.302141 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.302107    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"]
Oct 30 14:08:00.347562 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.347528    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bqbf\" (UniqueName: \"kubernetes.io/projected/118ff43b-bd6b-4f33-ba72-1590e2d3004c-kube-api-access-7bqbf\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.347764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.347564    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.347833 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.347803    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.347877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.347849    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.347933 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.347914    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz7p8\" (UniqueName: \"kubernetes.io/projected/fd30bd6f-7d43-4aee-83e1-ab992521822b-kube-api-access-tz7p8\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.347990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.347980    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.449353 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.449313    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bqbf\" (UniqueName: \"kubernetes.io/projected/118ff43b-bd6b-4f33-ba72-1590e2d3004c-kube-api-access-7bqbf\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.449353 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.449356    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.449598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.449538    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.449598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.449567    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.449598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.449592    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz7p8\" (UniqueName: \"kubernetes.io/projected/fd30bd6f-7d43-4aee-83e1-ab992521822b-kube-api-access-tz7p8\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.449843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.449677    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.450252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.450223    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.450411 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.450392    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.452093 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.452074    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.452341 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.452325    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.468800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.468767    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz7p8\" (UniqueName: \"kubernetes.io/projected/fd30bd6f-7d43-4aee-83e1-ab992521822b-kube-api-access-tz7p8\") pod \"maas-api-revoke-sub-6dc75877-trlrf\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.468800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.468792    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bqbf\" (UniqueName: \"kubernetes.io/projected/118ff43b-bd6b-4f33-ba72-1590e2d3004c-kube-api-access-7bqbf\") pod \"maas-api-revoke-sub-a7390182-wb76j\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.519133 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.519055    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"]
Oct 30 14:08:00.529012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.528974    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.542449 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.542416    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"]
Oct 30 14:08:00.565650 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.565606    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:00.589216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.589184    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:00.677828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.676442    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19790061-c881-4391-9adf-19927958106a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.677828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.676589    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf46p\" (UniqueName: \"kubernetes.io/projected/19790061-c881-4391-9adf-19927958106a-kube-api-access-sf46p\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.677828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.676901    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19790061-c881-4391-9adf-19927958106a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.740034 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.739785    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:08:00.778619 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.777976    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19790061-c881-4391-9adf-19927958106a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.778619 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.778038    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19790061-c881-4391-9adf-19927958106a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.778619 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.778294    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf46p\" (UniqueName: \"kubernetes.io/projected/19790061-c881-4391-9adf-19927958106a-kube-api-access-sf46p\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.779460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.779433    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19790061-c881-4391-9adf-19927958106a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.781432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.781402    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19790061-c881-4391-9adf-19927958106a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.797995 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.797966    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf46p\" (UniqueName: \"kubernetes.io/projected/19790061-c881-4391-9adf-19927958106a-kube-api-access-sf46p\") pod \"maas-api-revoke-sub-f873afaa-ldv87\" (UID: \"19790061-c881-4391-9adf-19927958106a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.846892 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.846859    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:00.890003 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.889967    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"]
Oct 30 14:08:00.891258 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:08:00.891227    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd30bd6f_7d43_4aee_83e1_ab992521822b.slice/crio-17cb98c60f1bba396473c432aa5705e79924f7d2df7f5927f92c9069f5bb7643 WatchSource:0}: Error finding container 17cb98c60f1bba396473c432aa5705e79924f7d2df7f5927f92c9069f5bb7643: Status 404 returned error can't find the container with id 17cb98c60f1bba396473c432aa5705e79924f7d2df7f5927f92c9069f5bb7643
Oct 30 14:08:00.930252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.930215    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf" event={"ID":"fd30bd6f-7d43-4aee-83e1-ab992521822b","Type":"ContainerStarted","Data":"17cb98c60f1bba396473c432aa5705e79924f7d2df7f5927f92c9069f5bb7643"}
Oct 30 14:08:00.981983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:00.981950    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"]
Oct 30 14:08:00.983364 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:08:00.983339    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod118ff43b_bd6b_4f33_ba72_1590e2d3004c.slice/crio-f2aa26bf0a5da5f96f8762ba53b7a4b5be43e505408941df957dbf99b1cc1c5a WatchSource:0}: Error finding container f2aa26bf0a5da5f96f8762ba53b7a4b5be43e505408941df957dbf99b1cc1c5a: Status 404 returned error can't find the container with id f2aa26bf0a5da5f96f8762ba53b7a4b5be43e505408941df957dbf99b1cc1c5a
Oct 30 14:08:01.155470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:01.155437    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"]
Oct 30 14:08:01.164555 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:08:01.164521    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19790061_c881_4391_9adf_19927958106a.slice/crio-7970ced75c682a98571000e9a9d3b7783bbf360923cdedfcfe9f6cb00b6d00ac WatchSource:0}: Error finding container 7970ced75c682a98571000e9a9d3b7783bbf360923cdedfcfe9f6cb00b6d00ac: Status 404 returned error can't find the container with id 7970ced75c682a98571000e9a9d3b7783bbf360923cdedfcfe9f6cb00b6d00ac
Oct 30 14:08:02.126728 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.125765    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87" event={"ID":"19790061-c881-4391-9adf-19927958106a","Type":"ContainerStarted","Data":"5591e560bfa66850af185937addde4a18aec937a4ad2158a1c61354d0801448b"}
Oct 30 14:08:02.126728 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.125893    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87" event={"ID":"19790061-c881-4391-9adf-19927958106a","Type":"ContainerStarted","Data":"7970ced75c682a98571000e9a9d3b7783bbf360923cdedfcfe9f6cb00b6d00ac"}
Oct 30 14:08:02.133974 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.133931    4291 generic.go:356] "Generic (PLEG): container finished" podID="118ff43b-bd6b-4f33-ba72-1590e2d3004c" containerID="d9287841899ccfa405e55bd5ff987959b30f942d7ae8b76f292614cff7c3370c" exitCode=0
Oct 30 14:08:02.134183 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.134058    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j" event={"ID":"118ff43b-bd6b-4f33-ba72-1590e2d3004c","Type":"ContainerDied","Data":"d9287841899ccfa405e55bd5ff987959b30f942d7ae8b76f292614cff7c3370c"}
Oct 30 14:08:02.134183 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.134124    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j" event={"ID":"118ff43b-bd6b-4f33-ba72-1590e2d3004c","Type":"ContainerStarted","Data":"f2aa26bf0a5da5f96f8762ba53b7a4b5be43e505408941df957dbf99b1cc1c5a"}
Oct 30 14:08:02.147713 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.146510    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd30bd6f-7d43-4aee-83e1-ab992521822b" containerID="874b6dc2e69f1e56732b22d3d752956c1df59d3e1c40b52cf6dcd12f5fb1b8cd" exitCode=0
Oct 30 14:08:02.147713 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.146590    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf" event={"ID":"fd30bd6f-7d43-4aee-83e1-ab992521822b","Type":"ContainerDied","Data":"874b6dc2e69f1e56732b22d3d752956c1df59d3e1c40b52cf6dcd12f5fb1b8cd"}
Oct 30 14:08:02.251652 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:02.251585    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87" podStartSLOduration=2.251570961 podStartE2EDuration="2.251570961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:08:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:08:02.1883418 +0000 UTC m=+3892.448580034" watchObservedRunningTime="2026-10-30 14:08:02.251570961 +0000 UTC m=+3892.511809195"
Oct 30 14:08:03.155147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.155060    4291 generic.go:356] "Generic (PLEG): container finished" podID="19790061-c881-4391-9adf-19927958106a" containerID="5591e560bfa66850af185937addde4a18aec937a4ad2158a1c61354d0801448b" exitCode=0
Oct 30 14:08:03.155590 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.155145    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87" event={"ID":"19790061-c881-4391-9adf-19927958106a","Type":"ContainerDied","Data":"5591e560bfa66850af185937addde4a18aec937a4ad2158a1c61354d0801448b"}
Oct 30 14:08:03.323436 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.323390    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:03.371570 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.371538    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:03.399230 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.399195    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-openshift-ca\") pod \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") "
Oct 30 14:08:03.399470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.399345    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/118ff43b-bd6b-4f33-ba72-1590e2d3004c-kube-api-access-7bqbf\" (UniqueName: \"kubernetes.io/projected/118ff43b-bd6b-4f33-ba72-1590e2d3004c-kube-api-access-7bqbf\") pod \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") "
Oct 30 14:08:03.399470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.399450    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-xks-ca\") pod \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\" (UID: \"118ff43b-bd6b-4f33-ba72-1590e2d3004c\") "
Oct 30 14:08:03.399682 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.399652    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-openshift-ca" pod "118ff43b-bd6b-4f33-ba72-1590e2d3004c" (UID: "118ff43b-bd6b-4f33-ba72-1590e2d3004c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:03.400103 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.400066    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:03.401705 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.401677    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-xks-ca" pod "118ff43b-bd6b-4f33-ba72-1590e2d3004c" (UID: "118ff43b-bd6b-4f33-ba72-1590e2d3004c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:03.401849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.401686    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/118ff43b-bd6b-4f33-ba72-1590e2d3004c-kube-api-access-7bqbf" pod "118ff43b-bd6b-4f33-ba72-1590e2d3004c" (UID: "118ff43b-bd6b-4f33-ba72-1590e2d3004c"). InnerVolumeSpecName "kube-api-access-7bqbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:03.501504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.501466    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd30bd6f-7d43-4aee-83e1-ab992521822b-kube-api-access-tz7p8\" (UniqueName: \"kubernetes.io/projected/fd30bd6f-7d43-4aee-83e1-ab992521822b-kube-api-access-tz7p8\") pod \"fd30bd6f-7d43-4aee-83e1-ab992521822b\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") "
Oct 30 14:08:03.501779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.501580    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-openshift-ca\") pod \"fd30bd6f-7d43-4aee-83e1-ab992521822b\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") "
Oct 30 14:08:03.501779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.501610    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-xks-ca\") pod \"fd30bd6f-7d43-4aee-83e1-ab992521822b\" (UID: \"fd30bd6f-7d43-4aee-83e1-ab992521822b\") "
Oct 30 14:08:03.502015 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.501982    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-openshift-ca" pod "fd30bd6f-7d43-4aee-83e1-ab992521822b" (UID: "fd30bd6f-7d43-4aee-83e1-ab992521822b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:03.502192 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.502177    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bqbf\" (UniqueName: \"kubernetes.io/projected/118ff43b-bd6b-4f33-ba72-1590e2d3004c-kube-api-access-7bqbf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:03.502240 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.502196    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/118ff43b-bd6b-4f33-ba72-1590e2d3004c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:03.502240 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.502205    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:03.503834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.503803    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-xks-ca" pod "fd30bd6f-7d43-4aee-83e1-ab992521822b" (UID: "fd30bd6f-7d43-4aee-83e1-ab992521822b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:03.503923 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.503857    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd30bd6f-7d43-4aee-83e1-ab992521822b-kube-api-access-tz7p8" pod "fd30bd6f-7d43-4aee-83e1-ab992521822b" (UID: "fd30bd6f-7d43-4aee-83e1-ab992521822b"). InnerVolumeSpecName "kube-api-access-tz7p8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:03.603957 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.603919    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tz7p8\" (UniqueName: \"kubernetes.io/projected/fd30bd6f-7d43-4aee-83e1-ab992521822b-kube-api-access-tz7p8\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:03.603957 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:03.603950    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fd30bd6f-7d43-4aee-83e1-ab992521822b-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:04.163873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.163777    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"
Oct 30 14:08:04.163873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.163796    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j" event={"ID":"118ff43b-bd6b-4f33-ba72-1590e2d3004c","Type":"ContainerDied","Data":"f2aa26bf0a5da5f96f8762ba53b7a4b5be43e505408941df957dbf99b1cc1c5a"}
Oct 30 14:08:04.163873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.163850    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2aa26bf0a5da5f96f8762ba53b7a4b5be43e505408941df957dbf99b1cc1c5a"
Oct 30 14:08:04.165347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.165319    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"
Oct 30 14:08:04.165492 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.165367    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf" event={"ID":"fd30bd6f-7d43-4aee-83e1-ab992521822b","Type":"ContainerDied","Data":"17cb98c60f1bba396473c432aa5705e79924f7d2df7f5927f92c9069f5bb7643"}
Oct 30 14:08:04.165492 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.165412    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="17cb98c60f1bba396473c432aa5705e79924f7d2df7f5927f92c9069f5bb7643"
Oct 30 14:08:04.399798 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.399766    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:04.532863 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.532762    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19790061-c881-4391-9adf-19927958106a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19790061-c881-4391-9adf-19927958106a-maas-api-xks-ca\") pod \"19790061-c881-4391-9adf-19927958106a\" (UID: \"19790061-c881-4391-9adf-19927958106a\") "
Oct 30 14:08:04.533068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.532863    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19790061-c881-4391-9adf-19927958106a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19790061-c881-4391-9adf-19927958106a-maas-api-openshift-ca\") pod \"19790061-c881-4391-9adf-19927958106a\" (UID: \"19790061-c881-4391-9adf-19927958106a\") "
Oct 30 14:08:04.533068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.532910    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19790061-c881-4391-9adf-19927958106a-kube-api-access-sf46p\" (UniqueName: \"kubernetes.io/projected/19790061-c881-4391-9adf-19927958106a-kube-api-access-sf46p\") pod \"19790061-c881-4391-9adf-19927958106a\" (UID: \"19790061-c881-4391-9adf-19927958106a\") "
Oct 30 14:08:04.533229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.533204    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19790061-c881-4391-9adf-19927958106a-maas-api-openshift-ca" pod "19790061-c881-4391-9adf-19927958106a" (UID: "19790061-c881-4391-9adf-19927958106a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:04.533570 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.533551    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19790061-c881-4391-9adf-19927958106a-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:04.535013 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.534995    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19790061-c881-4391-9adf-19927958106a-maas-api-xks-ca" pod "19790061-c881-4391-9adf-19927958106a" (UID: "19790061-c881-4391-9adf-19927958106a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:04.535168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.535154    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19790061-c881-4391-9adf-19927958106a-kube-api-access-sf46p" pod "19790061-c881-4391-9adf-19927958106a" (UID: "19790061-c881-4391-9adf-19927958106a"). InnerVolumeSpecName "kube-api-access-sf46p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:04.635105 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.635068    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19790061-c881-4391-9adf-19927958106a-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:04.635105 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:04.635096    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf46p\" (UniqueName: \"kubernetes.io/projected/19790061-c881-4391-9adf-19927958106a-kube-api-access-sf46p\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:05.173343 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:05.173312    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"
Oct 30 14:08:05.173343 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:05.173325    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87" event={"ID":"19790061-c881-4391-9adf-19927958106a","Type":"ContainerDied","Data":"7970ced75c682a98571000e9a9d3b7783bbf360923cdedfcfe9f6cb00b6d00ac"}
Oct 30 14:08:05.173829 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:05.173372    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7970ced75c682a98571000e9a9d3b7783bbf360923cdedfcfe9f6cb00b6d00ac"
Oct 30 14:08:08.212463 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:08:08.212424    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:08:11.460589 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:11.460550    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"]
Oct 30 14:08:11.472014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:11.471976    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-167965-hz9lm"]
Oct 30 14:08:11.879607 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:11.879561    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="343d1d3c-16c2-4a08-bf85-b27ef588c52b" path="/var/lib/kubelet/pods/343d1d3c-16c2-4a08-bf85-b27ef588c52b/volumes"
Oct 30 14:08:14.443567 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:14.443526    4291 scope.go:122] "RemoveContainer" containerID="540e88d468f2aab99d217ffe8a72a02d8affb454bfc5b8fa5b7006b20393fc6e"
Oct 30 14:08:14.455821 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:14.455790    4291 scope.go:122] "RemoveContainer" containerID="bbb555928d5525c3d395b64217abcb702e2efbf238bac0131ee4f785f7c7f701"
Oct 30 14:08:14.467233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:14.467208    4291 scope.go:122] "RemoveContainer" containerID="12d30dc58adfd15f446cc99cf2528dd4af3d565aac3417c9b83516144ce1f612"
Oct 30 14:08:18.968206 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:18.968166    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"]
Oct 30 14:08:18.970109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:18.970072    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy" containerID="cri-o://3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f" gracePeriod=30
Oct 30 14:08:19.352438 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.352405    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:08:19.525993 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.525954    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"]
Oct 30 14:08:19.526654 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526618    4291 cpu_manager.go:414] "containerMap: removing container" podUID="118ff43b-bd6b-4f33-ba72-1590e2d3004c" containerName="revoke-keys"
Oct 30 14:08:19.526654 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526653    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="118ff43b-bd6b-4f33-ba72-1590e2d3004c" containerName="revoke-keys"
Oct 30 14:08:19.526792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526670    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd30bd6f-7d43-4aee-83e1-ab992521822b" containerName="revoke-keys"
Oct 30 14:08:19.526792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526675    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd30bd6f-7d43-4aee-83e1-ab992521822b" containerName="revoke-keys"
Oct 30 14:08:19.526792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526705    4291 cpu_manager.go:414] "containerMap: removing container" podUID="19790061-c881-4391-9adf-19927958106a" containerName="revoke-keys"
Oct 30 14:08:19.526792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526710    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19790061-c881-4391-9adf-19927958106a" containerName="revoke-keys"
Oct 30 14:08:19.526925 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526823    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="118ff43b-bd6b-4f33-ba72-1590e2d3004c" containerName="revoke-keys"
Oct 30 14:08:19.526925 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526834    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd30bd6f-7d43-4aee-83e1-ab992521822b" containerName="revoke-keys"
Oct 30 14:08:19.526925 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.526845    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="19790061-c881-4391-9adf-19927958106a" containerName="revoke-keys"
Oct 30 14:08:19.530387 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.530363    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.545331 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.545309    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:08:19.546620 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.546590    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:08:19.549860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.549833    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"]
Oct 30 14:08:19.625545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.625466    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.625545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.625542    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.625772 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.625567    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmbcx\" (UniqueName: \"kubernetes.io/projected/baaa1ead-e982-4aff-9f77-02422a1acbbf-kube-api-access-mmbcx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.632029 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.632002    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:08:19.632029 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.632026    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:08:19.727231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.727194    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.727428 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.727261    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.727428 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.727286    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmbcx\" (UniqueName: \"kubernetes.io/projected/baaa1ead-e982-4aff-9f77-02422a1acbbf-kube-api-access-mmbcx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.727913 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.727890    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.729647 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.729616    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.746195 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.746172    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmbcx\" (UniqueName: \"kubernetes.io/projected/baaa1ead-e982-4aff-9f77-02422a1acbbf-kube-api-access-mmbcx\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:19.844929 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:19.844898    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:20.205460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:20.205434    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"]
Oct 30 14:08:20.206867 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:08:20.206841    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaa1ead_e982_4aff_9f77_02422a1acbbf.slice/crio-0427bd677fed186d5b65e318b9799d03216ddda4ea49a59064a0c0295902e7d7 WatchSource:0}: Error finding container 0427bd677fed186d5b65e318b9799d03216ddda4ea49a59064a0c0295902e7d7: Status 404 returned error can't find the container with id 0427bd677fed186d5b65e318b9799d03216ddda4ea49a59064a0c0295902e7d7
Oct 30 14:08:20.403189 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:20.401822    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm" event={"ID":"baaa1ead-e982-4aff-9f77-02422a1acbbf","Type":"ContainerStarted","Data":"0427bd677fed186d5b65e318b9799d03216ddda4ea49a59064a0c0295902e7d7"}
Oct 30 14:08:21.409849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:21.409813    4291 generic.go:356] "Generic (PLEG): container finished" podID="baaa1ead-e982-4aff-9f77-02422a1acbbf" containerID="02b7a0db07671d728af6ede7e3328d0922e0ab4c386be990e73810e280cca7ab" exitCode=0
Oct 30 14:08:21.410219 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:21.409863    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm" event={"ID":"baaa1ead-e982-4aff-9f77-02422a1acbbf","Type":"ContainerDied","Data":"02b7a0db07671d728af6ede7e3328d0922e0ab4c386be990e73810e280cca7ab"}
Oct 30 14:08:22.579147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.579123    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:22.714970 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.714896    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-openshift-ca\") pod \"baaa1ead-e982-4aff-9f77-02422a1acbbf\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") "
Oct 30 14:08:22.715115 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.714994    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-xks-ca\") pod \"baaa1ead-e982-4aff-9f77-02422a1acbbf\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") "
Oct 30 14:08:22.715115 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.715034    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baaa1ead-e982-4aff-9f77-02422a1acbbf-kube-api-access-mmbcx\" (UniqueName: \"kubernetes.io/projected/baaa1ead-e982-4aff-9f77-02422a1acbbf-kube-api-access-mmbcx\") pod \"baaa1ead-e982-4aff-9f77-02422a1acbbf\" (UID: \"baaa1ead-e982-4aff-9f77-02422a1acbbf\") "
Oct 30 14:08:22.715302 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.715277    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-openshift-ca" pod "baaa1ead-e982-4aff-9f77-02422a1acbbf" (UID: "baaa1ead-e982-4aff-9f77-02422a1acbbf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:22.715440 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.715424    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:22.717055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.717032    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-xks-ca" pod "baaa1ead-e982-4aff-9f77-02422a1acbbf" (UID: "baaa1ead-e982-4aff-9f77-02422a1acbbf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:08:22.717132 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.717116    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaa1ead-e982-4aff-9f77-02422a1acbbf-kube-api-access-mmbcx" pod "baaa1ead-e982-4aff-9f77-02422a1acbbf" (UID: "baaa1ead-e982-4aff-9f77-02422a1acbbf"). InnerVolumeSpecName "kube-api-access-mmbcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:22.816901 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.816841    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/baaa1ead-e982-4aff-9f77-02422a1acbbf-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:22.816901 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:22.816866    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmbcx\" (UniqueName: \"kubernetes.io/projected/baaa1ead-e982-4aff-9f77-02422a1acbbf-kube-api-access-mmbcx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:23.425250 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:23.425214    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"
Oct 30 14:08:23.425444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:23.425260    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm" event={"ID":"baaa1ead-e982-4aff-9f77-02422a1acbbf","Type":"ContainerDied","Data":"0427bd677fed186d5b65e318b9799d03216ddda4ea49a59064a0c0295902e7d7"}
Oct 30 14:08:23.425444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:23.425317    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0427bd677fed186d5b65e318b9799d03216ddda4ea49a59064a0c0295902e7d7"
Oct 30 14:08:24.133128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.133092    4291 patch_prober.go:29] interesting pod/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused" start-of-body=
Oct 30 14:08:24.133496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.133148    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.189:15021/healthz/ready\": dial tcp 10.133.0.189:15021: connect: connection refused"
Oct 30 14:08:24.244654 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.244589    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 14:08:24.341323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341287    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-token\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-token\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.341573 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341385    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-socket\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.341573 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341423    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-credential-socket\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.341573 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341530    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-envoy\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.341775 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341611    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istiod-ca-cert\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.341775 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341679    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-kube-api-access-w98qf\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-kube-api-access-w98qf\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.341775 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341704    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-socket" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:24.341775 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341762    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-certs\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.342012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341792    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-credential-socket" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:24.342012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341814    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-podinfo\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.342012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.341955    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-data\") pod \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\" (UID: \"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b\") "
Oct 30 14:08:24.342513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.342382    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istiod-ca-cert" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:24.342513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.342480    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-data" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:24.342844 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.342538    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-certs" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:24.342844 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.342650    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.342844 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.342668    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.344125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.344101    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-kube-api-access-w98qf" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "kube-api-access-w98qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:24.344241 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.344139    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-token" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:24.344312 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.344272    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-podinfo" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:08:24.344370 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.344351    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-envoy" pod "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" (UID: "f386f4e9-bf64-44f0-ba8d-d02e5966cb9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:24.433923 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.433896    4291 generic.go:356] "Generic (PLEG): container finished" podID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerID="3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f" exitCode=0
Oct 30 14:08:24.434085 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.433942    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"
Oct 30 14:08:24.434085 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.433983    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" event={"ID":"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b","Type":"ContainerDied","Data":"3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f"}
Oct 30 14:08:24.434085 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.434044    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk" event={"ID":"f386f4e9-bf64-44f0-ba8d-d02e5966cb9b","Type":"ContainerDied","Data":"25bb0a701c660c3d36c12b9f5414cce9d23f46c4d47bcfefd1a6e1cb2fb3fff3"}
Oct 30 14:08:24.434085 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.434072    4291 scope.go:122] "RemoveContainer" containerID="3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f"
Oct 30 14:08:24.445417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.445033    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.445417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.445056    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.445417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.445094    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w98qf\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-kube-api-access-w98qf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.445417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.445110    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.445417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.445125    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.445417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.445151    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.445417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.445168    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:24.452765 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.452746    4291 scope.go:122] "RemoveContainer" containerID="3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f"
Oct 30 14:08:24.453048 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:08:24.453025    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f\": container with ID starting with 3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f not found: ID does not exist" containerID="3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f"
Oct 30 14:08:24.453147 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.453052    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f"} err="failed to get container status \"3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f\": rpc error: code = NotFound desc = could not find container \"3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f\": container with ID starting with 3ea7b03088296752a707d1f1620af9633092a45136486346d6f5c737534a429f not found: ID does not exist"
Oct 30 14:08:24.485456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.485429    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"]
Oct 30 14:08:24.489989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:24.489875    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-167965-openshift-default-54bc9d6ddc-cmcwk"]
Oct 30 14:08:25.878918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:25.878885    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" path="/var/lib/kubelet/pods/f386f4e9-bf64-44f0-ba8d-d02e5966cb9b/volumes"
Oct 30 14:08:37.341905 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:37.341857    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"]
Oct 30 14:08:37.347229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:37.347195    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5852ff-sxwpn"]
Oct 30 14:08:37.879095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:37.879062    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83866d27-a6c4-4198-98ef-40817bab0941" path="/var/lib/kubelet/pods/83866d27-a6c4-4198-98ef-40817bab0941/volumes"
Oct 30 14:08:38.211688 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:08:38.211561    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:08:45.222813 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:45.222763    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"]
Oct 30 14:08:45.224222 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:45.224185    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" containerID="cri-o://1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c" gracePeriod=30
Oct 30 14:08:45.445525 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:45.445487    4291 patch_prober.go:29] interesting pod/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:08:45.445868 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:45.445829    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:08:50.097427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.097387    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:08:50.500715 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.500689    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 14:08:50.625180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625146    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-kube-api-access-5wb59\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-kube-api-access-5wb59\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625225    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-data\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625324    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-credential-socket\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625357    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-certs\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625396    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-token\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-token\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625449    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istiod-ca-cert\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625498    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-envoy\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625535    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-socket\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625575    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-data" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:50.625819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625593    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-podinfo\") pod \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\" (UID: \"e73a9c61-a0fd-4802-9864-cf2d8bde666b\") "
Oct 30 14:08:50.625819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625611    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-credential-socket" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:50.625819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625680    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-certs" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:50.626120 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625842    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-socket" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:50.626120 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.625908    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istiod-ca-cert" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:08:50.626249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.626151    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.626249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.626167    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.626249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.626175    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.626249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.626184    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.626249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.626194    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.627761 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.627732    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-podinfo" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:08:50.627910 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.627770    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-kube-api-access-5wb59" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "kube-api-access-5wb59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:50.628068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.628047    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-envoy" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:08:50.628151 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.628088    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-token" pod "e73a9c61-a0fd-4802-9864-cf2d8bde666b" (UID: "e73a9c61-a0fd-4802-9864-cf2d8bde666b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:08:50.692531 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.692433    4291 generic.go:356] "Generic (PLEG): container finished" podID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerID="1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c" exitCode=0
Oct 30 14:08:50.692531 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.692492    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"
Oct 30 14:08:50.692796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.692518    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" event={"ID":"e73a9c61-a0fd-4802-9864-cf2d8bde666b","Type":"ContainerDied","Data":"1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c"}
Oct 30 14:08:50.692796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.692578    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz" event={"ID":"e73a9c61-a0fd-4802-9864-cf2d8bde666b","Type":"ContainerDied","Data":"38d3ebd4a71fe180c1d4fc45bce886a1aa955ff7ec5a0fd1d90e3378216d97cf"}
Oct 30 14:08:50.692796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.692602    4291 scope.go:122] "RemoveContainer" containerID="1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c"
Oct 30 14:08:50.704485 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.704460    4291 scope.go:122] "RemoveContainer" containerID="1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c"
Oct 30 14:08:50.704823 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:08:50.704803    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c\": container with ID starting with 1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c not found: ID does not exist" containerID="1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c"
Oct 30 14:08:50.704891 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.704833    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c"} err="failed to get container status \"1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c\": rpc error: code = NotFound desc = could not find container \"1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c\": container with ID starting with 1b0b5ed9405f7f955332fcb5dd05e6a289fc936887d5e003adcf60ee8b396a1c not found: ID does not exist"
Oct 30 14:08:50.727180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.727152    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.727180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.727180    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.727361 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.727192    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5wb59\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-kube-api-access-5wb59\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.727361 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.727206    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e73a9c61-a0fd-4802-9864-cf2d8bde666b-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:08:50.742737 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.742710    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"]
Oct 30 14:08:50.746834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:50.746809    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5852ff-openshift-default-d8cf4cc4c-tf8zz"]
Oct 30 14:08:51.878379 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:51.878347    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" path="/var/lib/kubelet/pods/e73a9c61-a0fd-4802-9864-cf2d8bde666b/volumes"
Oct 30 14:08:57.290517 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:57.290478    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"]
Oct 30 14:08:57.301913 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:57.301886    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f12b80-6sjlm"]
Oct 30 14:08:57.879170 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:08:57.879137    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baaa1ead-e982-4aff-9f77-02422a1acbbf" path="/var/lib/kubelet/pods/baaa1ead-e982-4aff-9f77-02422a1acbbf/volumes"
Oct 30 14:09:05.440227 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440184    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"]
Oct 30 14:09:05.440784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440769    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy"
Oct 30 14:09:05.440835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440784    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy"
Oct 30 14:09:05.440835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440796    4291 cpu_manager.go:414] "containerMap: removing container" podUID="baaa1ead-e982-4aff-9f77-02422a1acbbf" containerName="revoke-keys"
Oct 30 14:09:05.440835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440803    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baaa1ead-e982-4aff-9f77-02422a1acbbf" containerName="revoke-keys"
Oct 30 14:09:05.440835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440814    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy"
Oct 30 14:09:05.440835 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440818    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy"
Oct 30 14:09:05.441006 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440911    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f386f4e9-bf64-44f0-ba8d-d02e5966cb9b" containerName="istio-proxy"
Oct 30 14:09:05.441006 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440920    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="baaa1ead-e982-4aff-9f77-02422a1acbbf" containerName="revoke-keys"
Oct 30 14:09:05.441006 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.440934    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e73a9c61-a0fd-4802-9864-cf2d8bde666b" containerName="istio-proxy"
Oct 30 14:09:05.473708 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.464462    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.510667 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.495925    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:05.510667 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.497814    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:09:05.510667 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.497945    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:09:05.510667 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.499649    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:09:05.518196 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.518159    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.518547 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.518515    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-868qk\" (UniqueName: \"kubernetes.io/projected/b9ec538a-6816-4d57-89f0-1db82498e499-kube-api-access-868qk\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.518829 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.518808    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.524019 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.523993    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"]
Oct 30 14:09:05.620726 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.620689    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.620905 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.620742    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-868qk\" (UniqueName: \"kubernetes.io/projected/b9ec538a-6816-4d57-89f0-1db82498e499-kube-api-access-868qk\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.620905 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.620824    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.621455 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.621434    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.623224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.623196    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.639969 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.639936    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-868qk\" (UniqueName: \"kubernetes.io/projected/b9ec538a-6816-4d57-89f0-1db82498e499-kube-api-access-868qk\") pod \"maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:05.816185 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:05.816097    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:06.123552 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:06.123520    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"]
Oct 30 14:09:06.126108 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:09:06.126068    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9ec538a_6816_4d57_89f0_1db82498e499.slice/crio-14808100dc3efc8361edb16d0cc4330dd0da41875ac63c6697698f1a8dee92ef WatchSource:0}: Error finding container 14808100dc3efc8361edb16d0cc4330dd0da41875ac63c6697698f1a8dee92ef: Status 404 returned error can't find the container with id 14808100dc3efc8361edb16d0cc4330dd0da41875ac63c6697698f1a8dee92ef
Oct 30 14:09:06.846309 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:06.846274    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7" event={"ID":"b9ec538a-6816-4d57-89f0-1db82498e499","Type":"ContainerStarted","Data":"25e4e09beb85365b42f137a562a9807db9433f7598c449a15a037ce81a9d8a0c"}
Oct 30 14:09:06.846714 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:06.846328    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7" event={"ID":"b9ec538a-6816-4d57-89f0-1db82498e499","Type":"ContainerStarted","Data":"14808100dc3efc8361edb16d0cc4330dd0da41875ac63c6697698f1a8dee92ef"}
Oct 30 14:09:06.887324 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:06.887272    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7" podStartSLOduration=1.887257991 podStartE2EDuration="1.887257991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:06.884325361 +0000 UTC m=+3957.144563597" watchObservedRunningTime="2026-10-30 14:09:06.887257991 +0000 UTC m=+3957.147496225"
Oct 30 14:09:07.854745 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:07.854712    4291 generic.go:356] "Generic (PLEG): container finished" podID="b9ec538a-6816-4d57-89f0-1db82498e499" containerID="25e4e09beb85365b42f137a562a9807db9433f7598c449a15a037ce81a9d8a0c" exitCode=0
Oct 30 14:09:07.855148 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:07.854802    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7" event={"ID":"b9ec538a-6816-4d57-89f0-1db82498e499","Type":"ContainerDied","Data":"25e4e09beb85365b42f137a562a9807db9433f7598c449a15a037ce81a9d8a0c"}
Oct 30 14:09:08.210577 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:09:08.210483    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:09:09.047960 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.047932    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:09.163272 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.163194    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-xks-ca\") pod \"b9ec538a-6816-4d57-89f0-1db82498e499\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") "
Oct 30 14:09:09.163272 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.163253    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-openshift-ca\") pod \"b9ec538a-6816-4d57-89f0-1db82498e499\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") "
Oct 30 14:09:09.163520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.163368    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9ec538a-6816-4d57-89f0-1db82498e499-kube-api-access-868qk\" (UniqueName: \"kubernetes.io/projected/b9ec538a-6816-4d57-89f0-1db82498e499-kube-api-access-868qk\") pod \"b9ec538a-6816-4d57-89f0-1db82498e499\" (UID: \"b9ec538a-6816-4d57-89f0-1db82498e499\") "
Oct 30 14:09:09.163700 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.163673    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-openshift-ca" pod "b9ec538a-6816-4d57-89f0-1db82498e499" (UID: "b9ec538a-6816-4d57-89f0-1db82498e499"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:09.163853 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.163836    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:09.165442 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.165420    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-xks-ca" pod "b9ec538a-6816-4d57-89f0-1db82498e499" (UID: "b9ec538a-6816-4d57-89f0-1db82498e499"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:09.165536 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.165500    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9ec538a-6816-4d57-89f0-1db82498e499-kube-api-access-868qk" pod "b9ec538a-6816-4d57-89f0-1db82498e499" (UID: "b9ec538a-6816-4d57-89f0-1db82498e499"). InnerVolumeSpecName "kube-api-access-868qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:09.265210 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.265180    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-868qk\" (UniqueName: \"kubernetes.io/projected/b9ec538a-6816-4d57-89f0-1db82498e499-kube-api-access-868qk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:09.265210 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.265207    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9ec538a-6816-4d57-89f0-1db82498e499-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:09.874154 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.874123    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"
Oct 30 14:09:09.878017 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.877990    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7" event={"ID":"b9ec538a-6816-4d57-89f0-1db82498e499","Type":"ContainerDied","Data":"14808100dc3efc8361edb16d0cc4330dd0da41875ac63c6697698f1a8dee92ef"}
Oct 30 14:09:09.878183 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:09.878031    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14808100dc3efc8361edb16d0cc4330dd0da41875ac63c6697698f1a8dee92ef"
Oct 30 14:09:14.533712 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:14.533678    4291 scope.go:122] "RemoveContainer" containerID="e198717ae8a55340c85f344d99ae2a61f1b433b1653b180b412552bad355c4e9"
Oct 30 14:09:14.543526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:14.543505    4291 scope.go:122] "RemoveContainer" containerID="d887cbdb095730d363a68010249dd75c4c28a49e837ad053ce32a6761436019d"
Oct 30 14:09:14.552358 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:14.552339    4291 scope.go:122] "RemoveContainer" containerID="e7de6db1474ed169d74c6bfff7a985473b8593e98f0c5bce7837de86d952d1c8"
Oct 30 14:09:14.560507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:14.560487    4291 scope.go:122] "RemoveContainer" containerID="87f6b0cc8df45121b52c7de62e216aa7589cb94de24dae75497129f7b0cf9222"
Oct 30 14:09:22.154021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.153990    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"]
Oct 30 14:09:22.154642 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.154618    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b9ec538a-6816-4d57-89f0-1db82498e499" containerName="revoke-keys"
Oct 30 14:09:22.154791 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.154650    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9ec538a-6816-4d57-89f0-1db82498e499" containerName="revoke-keys"
Oct 30 14:09:22.154791 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.154783    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9ec538a-6816-4d57-89f0-1db82498e499" containerName="revoke-keys"
Oct 30 14:09:22.158482 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.158463    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.170966 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.170945    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"]
Oct 30 14:09:22.174997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.174979    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"]
Oct 30 14:09:22.175103 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.175062    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.178950 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.178928    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:09:22.179083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.178956    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:22.179083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.178978    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:09:22.181971 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.181951    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:09:22.189916 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.189892    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"]
Oct 30 14:09:22.239589 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.239548    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5q5q\" (UniqueName: \"kubernetes.io/projected/e57a8441-6a8e-4cc3-9831-499ad518296f-kube-api-access-d5q5q\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.239788 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.239619    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.239788 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.239740    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.239788 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.239778    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5gb7\" (UniqueName: \"kubernetes.io/projected/c8b43b30-c497-4c3a-935e-e5e7d1914313-kube-api-access-r5gb7\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.240012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.239993    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.240055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.240038    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.340925 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.340885    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.341143 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.340940    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.341143 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.340969    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5q5q\" (UniqueName: \"kubernetes.io/projected/e57a8441-6a8e-4cc3-9831-499ad518296f-kube-api-access-d5q5q\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.341143 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.341116    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.341335 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.341169    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.341335 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.341217    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5gb7\" (UniqueName: \"kubernetes.io/projected/c8b43b30-c497-4c3a-935e-e5e7d1914313-kube-api-access-r5gb7\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.341624 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.341599    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.341883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.341864    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.343440 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.343413    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.343720 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.343697    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.360569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.360543    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5gb7\" (UniqueName: \"kubernetes.io/projected/c8b43b30-c497-4c3a-935e-e5e7d1914313-kube-api-access-r5gb7\") pod \"maas-api-revoke-sub-fa4e7527-vmz45\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.360680 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.360545    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5q5q\" (UniqueName: \"kubernetes.io/projected/e57a8441-6a8e-4cc3-9831-499ad518296f-kube-api-access-d5q5q\") pod \"maas-api-revoke-sub-98c1547d-46rxs\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.473172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.473095    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:22.494460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.494434    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:22.777021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.776992    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"]
Oct 30 14:09:22.778739 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:09:22.778714    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b43b30_c497_4c3a_935e_e5e7d1914313.slice/crio-c17c1ad997715da60834c73b0726a8b34b4e37702891ba0a4f8e6109d1af72a0 WatchSource:0}: Error finding container c17c1ad997715da60834c73b0726a8b34b4e37702891ba0a4f8e6109d1af72a0: Status 404 returned error can't find the container with id c17c1ad997715da60834c73b0726a8b34b4e37702891ba0a4f8e6109d1af72a0
Oct 30 14:09:22.810321 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.810296    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"]
Oct 30 14:09:22.811509 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:09:22.811480    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode57a8441_6a8e_4cc3_9831_499ad518296f.slice/crio-a05e344ae9a1f4784809b6e443681c2f0fdfe5fa10113ddd7d60dee10f78e0fc WatchSource:0}: Error finding container a05e344ae9a1f4784809b6e443681c2f0fdfe5fa10113ddd7d60dee10f78e0fc: Status 404 returned error can't find the container with id a05e344ae9a1f4784809b6e443681c2f0fdfe5fa10113ddd7d60dee10f78e0fc
Oct 30 14:09:22.959152 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.959116    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45" event={"ID":"c8b43b30-c497-4c3a-935e-e5e7d1914313","Type":"ContainerStarted","Data":"c17c1ad997715da60834c73b0726a8b34b4e37702891ba0a4f8e6109d1af72a0"}
Oct 30 14:09:22.960325 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:22.960305    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs" event={"ID":"e57a8441-6a8e-4cc3-9831-499ad518296f","Type":"ContainerStarted","Data":"a05e344ae9a1f4784809b6e443681c2f0fdfe5fa10113ddd7d60dee10f78e0fc"}
Oct 30 14:09:23.968180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:23.968085    4291 generic.go:356] "Generic (PLEG): container finished" podID="c8b43b30-c497-4c3a-935e-e5e7d1914313" containerID="e2bf3612aba56b4bdf761853906bd3972fd9d7ad0f76fc64547009200de5bf64" exitCode=0
Oct 30 14:09:23.968180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:23.968164    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45" event={"ID":"c8b43b30-c497-4c3a-935e-e5e7d1914313","Type":"ContainerDied","Data":"e2bf3612aba56b4bdf761853906bd3972fd9d7ad0f76fc64547009200de5bf64"}
Oct 30 14:09:23.969844 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:23.969824    4291 generic.go:356] "Generic (PLEG): container finished" podID="e57a8441-6a8e-4cc3-9831-499ad518296f" containerID="f3892495ba375a25d0f64ce6f279ad291778ead8fb79f49e96971908f27df2f7" exitCode=0
Oct 30 14:09:23.969926 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:23.969907    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs" event={"ID":"e57a8441-6a8e-4cc3-9831-499ad518296f","Type":"ContainerDied","Data":"f3892495ba375a25d0f64ce6f279ad291778ead8fb79f49e96971908f27df2f7"}
Oct 30 14:09:25.127384 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.127359    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:25.170983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.170956    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:25.283893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.283806    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-xks-ca\") pod \"c8b43b30-c497-4c3a-935e-e5e7d1914313\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") "
Oct 30 14:09:25.283893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.283872    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-openshift-ca\") pod \"c8b43b30-c497-4c3a-935e-e5e7d1914313\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") "
Oct 30 14:09:25.284126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.283907    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-xks-ca\") pod \"e57a8441-6a8e-4cc3-9831-499ad518296f\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") "
Oct 30 14:09:25.284126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.283951    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-openshift-ca\") pod \"e57a8441-6a8e-4cc3-9831-499ad518296f\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") "
Oct 30 14:09:25.284126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.283988    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b43b30-c497-4c3a-935e-e5e7d1914313-kube-api-access-r5gb7\" (UniqueName: \"kubernetes.io/projected/c8b43b30-c497-4c3a-935e-e5e7d1914313-kube-api-access-r5gb7\") pod \"c8b43b30-c497-4c3a-935e-e5e7d1914313\" (UID: \"c8b43b30-c497-4c3a-935e-e5e7d1914313\") "
Oct 30 14:09:25.284126 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.284060    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e57a8441-6a8e-4cc3-9831-499ad518296f-kube-api-access-d5q5q\" (UniqueName: \"kubernetes.io/projected/e57a8441-6a8e-4cc3-9831-499ad518296f-kube-api-access-d5q5q\") pod \"e57a8441-6a8e-4cc3-9831-499ad518296f\" (UID: \"e57a8441-6a8e-4cc3-9831-499ad518296f\") "
Oct 30 14:09:25.284359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.284298    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-openshift-ca" pod "c8b43b30-c497-4c3a-935e-e5e7d1914313" (UID: "c8b43b30-c497-4c3a-935e-e5e7d1914313"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:25.284421 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.284376    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-openshift-ca" pod "e57a8441-6a8e-4cc3-9831-499ad518296f" (UID: "e57a8441-6a8e-4cc3-9831-499ad518296f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:25.284655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.284619    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.284800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.284667    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.286305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.286281    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b43b30-c497-4c3a-935e-e5e7d1914313-kube-api-access-r5gb7" pod "c8b43b30-c497-4c3a-935e-e5e7d1914313" (UID: "c8b43b30-c497-4c3a-935e-e5e7d1914313"). InnerVolumeSpecName "kube-api-access-r5gb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:25.286402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.286309    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e57a8441-6a8e-4cc3-9831-499ad518296f-kube-api-access-d5q5q" pod "e57a8441-6a8e-4cc3-9831-499ad518296f" (UID: "e57a8441-6a8e-4cc3-9831-499ad518296f"). InnerVolumeSpecName "kube-api-access-d5q5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:25.286465 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.286453    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-xks-ca" pod "e57a8441-6a8e-4cc3-9831-499ad518296f" (UID: "e57a8441-6a8e-4cc3-9831-499ad518296f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:25.286549 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.286534    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-xks-ca" pod "c8b43b30-c497-4c3a-935e-e5e7d1914313" (UID: "c8b43b30-c497-4c3a-935e-e5e7d1914313"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:25.385742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.385701    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b43b30-c497-4c3a-935e-e5e7d1914313-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.385742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.385739    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e57a8441-6a8e-4cc3-9831-499ad518296f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.385931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.385754    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5gb7\" (UniqueName: \"kubernetes.io/projected/c8b43b30-c497-4c3a-935e-e5e7d1914313-kube-api-access-r5gb7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.385931 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.385768    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5q5q\" (UniqueName: \"kubernetes.io/projected/e57a8441-6a8e-4cc3-9831-499ad518296f-kube-api-access-d5q5q\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:25.985463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.985432    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"
Oct 30 14:09:25.985671 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.985467    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs" event={"ID":"e57a8441-6a8e-4cc3-9831-499ad518296f","Type":"ContainerDied","Data":"a05e344ae9a1f4784809b6e443681c2f0fdfe5fa10113ddd7d60dee10f78e0fc"}
Oct 30 14:09:25.985671 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.985517    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05e344ae9a1f4784809b6e443681c2f0fdfe5fa10113ddd7d60dee10f78e0fc"
Oct 30 14:09:25.987101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.987072    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45" event={"ID":"c8b43b30-c497-4c3a-935e-e5e7d1914313","Type":"ContainerDied","Data":"c17c1ad997715da60834c73b0726a8b34b4e37702891ba0a4f8e6109d1af72a0"}
Oct 30 14:09:25.987218 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.987125    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c17c1ad997715da60834c73b0726a8b34b4e37702891ba0a4f8e6109d1af72a0"
Oct 30 14:09:25.987218 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:25.987084    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"
Oct 30 14:09:32.949984 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:32.949951    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:09:37.031215 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:37.031180    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"]
Oct 30 14:09:37.042750 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:37.042716    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-73641705-lzdjc"]
Oct 30 14:09:37.879069 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:37.879033    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0" path="/var/lib/kubelet/pods/9f9c7fac-d05f-4f32-bc4f-e1fe45c115f0/volumes"
Oct 30 14:09:38.210484 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:09:38.210399    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:09:41.297278 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.297244    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"]
Oct 30 14:09:41.297819 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.297804    4291 cpu_manager.go:414] "containerMap: removing container" podUID="e57a8441-6a8e-4cc3-9831-499ad518296f" containerName="revoke-keys"
Oct 30 14:09:41.297870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.297820    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e57a8441-6a8e-4cc3-9831-499ad518296f" containerName="revoke-keys"
Oct 30 14:09:41.297870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.297841    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c8b43b30-c497-4c3a-935e-e5e7d1914313" containerName="revoke-keys"
Oct 30 14:09:41.297870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.297846    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b43b30-c497-4c3a-935e-e5e7d1914313" containerName="revoke-keys"
Oct 30 14:09:41.298628 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.298600    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="e57a8441-6a8e-4cc3-9831-499ad518296f" containerName="revoke-keys"
Oct 30 14:09:41.298838 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.298823    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b43b30-c497-4c3a-935e-e5e7d1914313" containerName="revoke-keys"
Oct 30 14:09:41.306067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.306043    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.319381 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.319351    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"]
Oct 30 14:09:41.325400 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.325372    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:41.325627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.325590    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:09:41.325627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.325617    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:09:41.332692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.332666    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:09:41.405929 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.405897    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.406101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.406061    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6pzn\" (UniqueName: \"kubernetes.io/projected/d90dd52e-32c1-4e08-9795-05717432a54c-kube-api-access-h6pzn\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.406101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.406093    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.507091 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.507052    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.507260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.507190    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6pzn\" (UniqueName: \"kubernetes.io/projected/d90dd52e-32c1-4e08-9795-05717432a54c-kube-api-access-h6pzn\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.507260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.507216    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.507840 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.507809    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.509542 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.509518    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.525604 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.525583    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6pzn\" (UniqueName: \"kubernetes.io/projected/d90dd52e-32c1-4e08-9795-05717432a54c-kube-api-access-h6pzn\") pod \"maas-api-revoke-sub-fa8454ad-4ckqt\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.626268 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.626238    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:41.962200 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:41.962117    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"]
Oct 30 14:09:41.965380 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:09:41.965350    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90dd52e_32c1_4e08_9795_05717432a54c.slice/crio-46d2897cd376b7d994ed5b63bcb60533c68d7c81ebf1ada98aa2b14eb80203e3 WatchSource:0}: Error finding container 46d2897cd376b7d994ed5b63bcb60533c68d7c81ebf1ada98aa2b14eb80203e3: Status 404 returned error can't find the container with id 46d2897cd376b7d994ed5b63bcb60533c68d7c81ebf1ada98aa2b14eb80203e3
Oct 30 14:09:42.102541 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:42.102509    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt" event={"ID":"d90dd52e-32c1-4e08-9795-05717432a54c","Type":"ContainerStarted","Data":"46d2897cd376b7d994ed5b63bcb60533c68d7c81ebf1ada98aa2b14eb80203e3"}
Oct 30 14:09:43.113914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:43.113881    4291 generic.go:356] "Generic (PLEG): container finished" podID="d90dd52e-32c1-4e08-9795-05717432a54c" containerID="6f54a9c3a7f4f309578735f6668731ab3eb8e6b52f8057bf673230061ab7a859" exitCode=0
Oct 30 14:09:43.114308 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:43.113928    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt" event={"ID":"d90dd52e-32c1-4e08-9795-05717432a54c","Type":"ContainerDied","Data":"6f54a9c3a7f4f309578735f6668731ab3eb8e6b52f8057bf673230061ab7a859"}
Oct 30 14:09:44.306756 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.306731    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:44.458278 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.458193    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90dd52e-32c1-4e08-9795-05717432a54c-kube-api-access-h6pzn\" (UniqueName: \"kubernetes.io/projected/d90dd52e-32c1-4e08-9795-05717432a54c-kube-api-access-h6pzn\") pod \"d90dd52e-32c1-4e08-9795-05717432a54c\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") "
Oct 30 14:09:44.458278 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.458245    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-openshift-ca\") pod \"d90dd52e-32c1-4e08-9795-05717432a54c\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") "
Oct 30 14:09:44.458483 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.458387    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-xks-ca\") pod \"d90dd52e-32c1-4e08-9795-05717432a54c\" (UID: \"d90dd52e-32c1-4e08-9795-05717432a54c\") "
Oct 30 14:09:44.458693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.458668    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-openshift-ca" pod "d90dd52e-32c1-4e08-9795-05717432a54c" (UID: "d90dd52e-32c1-4e08-9795-05717432a54c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:44.458912 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.458897    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:44.460428 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.460397    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-xks-ca" pod "d90dd52e-32c1-4e08-9795-05717432a54c" (UID: "d90dd52e-32c1-4e08-9795-05717432a54c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:44.460520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.460486    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90dd52e-32c1-4e08-9795-05717432a54c-kube-api-access-h6pzn" pod "d90dd52e-32c1-4e08-9795-05717432a54c" (UID: "d90dd52e-32c1-4e08-9795-05717432a54c"). InnerVolumeSpecName "kube-api-access-h6pzn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:44.560211 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.560170    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6pzn\" (UniqueName: \"kubernetes.io/projected/d90dd52e-32c1-4e08-9795-05717432a54c-kube-api-access-h6pzn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:44.560211 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:44.560204    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d90dd52e-32c1-4e08-9795-05717432a54c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:45.131056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:45.131023    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt" event={"ID":"d90dd52e-32c1-4e08-9795-05717432a54c","Type":"ContainerDied","Data":"46d2897cd376b7d994ed5b63bcb60533c68d7c81ebf1ada98aa2b14eb80203e3"}
Oct 30 14:09:45.131235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:45.131074    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46d2897cd376b7d994ed5b63bcb60533c68d7c81ebf1ada98aa2b14eb80203e3"
Oct 30 14:09:45.131235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:45.131034    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"
Oct 30 14:09:51.705684 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.705651    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"]
Oct 30 14:09:51.706169 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.706156    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d90dd52e-32c1-4e08-9795-05717432a54c" containerName="revoke-keys"
Oct 30 14:09:51.706220 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.706170    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90dd52e-32c1-4e08-9795-05717432a54c" containerName="revoke-keys"
Oct 30 14:09:51.706285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.706276    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90dd52e-32c1-4e08-9795-05717432a54c" containerName="revoke-keys"
Oct 30 14:09:51.709800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.709781    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.724587 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.724563    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:09:51.730442 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.730413    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"]
Oct 30 14:09:51.736089 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.736048    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:09:51.741504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.737501    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:09:51.741504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.737593    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:09:51.893651 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.893601    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.893848 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.893771    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.893923 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.893889    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krkgb\" (UniqueName: \"kubernetes.io/projected/7bf90d54-de4e-40f3-8995-08c2610e6172-kube-api-access-krkgb\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.994543 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.994472    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.994725 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.994587    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krkgb\" (UniqueName: \"kubernetes.io/projected/7bf90d54-de4e-40f3-8995-08c2610e6172-kube-api-access-krkgb\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.994815 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.994758    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.995425 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.995398    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:51.996873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:51.996852    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:52.013951 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:52.013933    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krkgb\" (UniqueName: \"kubernetes.io/projected/7bf90d54-de4e-40f3-8995-08c2610e6172-kube-api-access-krkgb\") pod \"maas-api-revoke-sub-2345757d-p4tz6\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:52.023736 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:52.023717    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:52.323711 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:52.323681    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"]
Oct 30 14:09:52.325139 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:09:52.325116    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bf90d54_de4e_40f3_8995_08c2610e6172.slice/crio-98621d4aba1515b3a2b77fe9e2c79d3fce8b0d27e488e1cbc6045b209ca10270 WatchSource:0}: Error finding container 98621d4aba1515b3a2b77fe9e2c79d3fce8b0d27e488e1cbc6045b209ca10270: Status 404 returned error can't find the container with id 98621d4aba1515b3a2b77fe9e2c79d3fce8b0d27e488e1cbc6045b209ca10270
Oct 30 14:09:53.210041 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:53.210002    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6" event={"ID":"7bf90d54-de4e-40f3-8995-08c2610e6172","Type":"ContainerStarted","Data":"ba339770320e5f98e26d3aff3dbbc538e29a6161ac78156b4656b38619003f1d"}
Oct 30 14:09:53.210490 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:53.210068    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6" event={"ID":"7bf90d54-de4e-40f3-8995-08c2610e6172","Type":"ContainerStarted","Data":"98621d4aba1515b3a2b77fe9e2c79d3fce8b0d27e488e1cbc6045b209ca10270"}
Oct 30 14:09:53.252558 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:53.252507    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6" podStartSLOduration=2.252493291 podStartE2EDuration="2.252493291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:09:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:09:53.250301205 +0000 UTC m=+4003.510539441" watchObservedRunningTime="2026-10-30 14:09:53.252493291 +0000 UTC m=+4003.512731524"
Oct 30 14:09:54.319182 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:54.318552    4291 generic.go:356] "Generic (PLEG): container finished" podID="7bf90d54-de4e-40f3-8995-08c2610e6172" containerID="ba339770320e5f98e26d3aff3dbbc538e29a6161ac78156b4656b38619003f1d" exitCode=0
Oct 30 14:09:54.319182 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:54.318816    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6" event={"ID":"7bf90d54-de4e-40f3-8995-08c2610e6172","Type":"ContainerDied","Data":"ba339770320e5f98e26d3aff3dbbc538e29a6161ac78156b4656b38619003f1d"}
Oct 30 14:09:54.508348 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:54.508318    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"]
Oct 30 14:09:54.513109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:54.513081    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-9bf8ad-px8p7"]
Oct 30 14:09:55.472272 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.472250    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:55.562463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.562415    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bf90d54-de4e-40f3-8995-08c2610e6172-kube-api-access-krkgb\" (UniqueName: \"kubernetes.io/projected/7bf90d54-de4e-40f3-8995-08c2610e6172-kube-api-access-krkgb\") pod \"7bf90d54-de4e-40f3-8995-08c2610e6172\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") "
Oct 30 14:09:55.562672 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.562499    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-openshift-ca\") pod \"7bf90d54-de4e-40f3-8995-08c2610e6172\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") "
Oct 30 14:09:55.562726 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.562675    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-xks-ca\") pod \"7bf90d54-de4e-40f3-8995-08c2610e6172\" (UID: \"7bf90d54-de4e-40f3-8995-08c2610e6172\") "
Oct 30 14:09:55.562899 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.562877    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-openshift-ca" pod "7bf90d54-de4e-40f3-8995-08c2610e6172" (UID: "7bf90d54-de4e-40f3-8995-08c2610e6172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:09:55.563071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.563058    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:55.564666 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.564611    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bf90d54-de4e-40f3-8995-08c2610e6172-kube-api-access-krkgb" pod "7bf90d54-de4e-40f3-8995-08c2610e6172" (UID: "7bf90d54-de4e-40f3-8995-08c2610e6172"). InnerVolumeSpecName "kube-api-access-krkgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:09:55.564790 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.564702    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-xks-ca" pod "7bf90d54-de4e-40f3-8995-08c2610e6172" (UID: "7bf90d54-de4e-40f3-8995-08c2610e6172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:09:55.664545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.664506    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bf90d54-de4e-40f3-8995-08c2610e6172-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:55.664545 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.664541    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krkgb\" (UniqueName: \"kubernetes.io/projected/7bf90d54-de4e-40f3-8995-08c2610e6172-kube-api-access-krkgb\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:09:55.878776 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:55.878740    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9ec538a-6816-4d57-89f0-1db82498e499" path="/var/lib/kubelet/pods/b9ec538a-6816-4d57-89f0-1db82498e499/volumes"
Oct 30 14:09:56.339170 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:56.339136    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"
Oct 30 14:09:56.339352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:56.339146    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6" event={"ID":"7bf90d54-de4e-40f3-8995-08c2610e6172","Type":"ContainerDied","Data":"98621d4aba1515b3a2b77fe9e2c79d3fce8b0d27e488e1cbc6045b209ca10270"}
Oct 30 14:09:56.339352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:09:56.339289    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98621d4aba1515b3a2b77fe9e2c79d3fce8b0d27e488e1cbc6045b209ca10270"
Oct 30 14:10:01.710215 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:01.710170    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"]
Oct 30 14:10:01.710709 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:01.710444    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy" containerID="cri-o://71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7" gracePeriod=30
Oct 30 14:10:06.982391 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:06.982325    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 14:10:07.154834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.154801    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-certs\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155008 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.154921    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-istio-token\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-istio-token\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155008 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.154976    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-socket\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155008 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155004    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-envoy\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155142 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155041    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-kube-api-access-gpfg2\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-kube-api-access-gpfg2\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155142 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155088    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-data\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155142 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155111    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-certs" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:07.155142 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155130    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/eed52cc6-2abf-47e5-979c-feadab32d548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed52cc6-2abf-47e5-979c-feadab32d548-istio-podinfo\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155160    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eed52cc6-2abf-47e5-979c-feadab32d548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed52cc6-2abf-47e5-979c-feadab32d548-istiod-ca-cert\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155220    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-credential-socket\") pod \"eed52cc6-2abf-47e5-979c-feadab32d548\" (UID: \"eed52cc6-2abf-47e5-979c-feadab32d548\") "
Oct 30 14:10:07.155568 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155540    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eed52cc6-2abf-47e5-979c-feadab32d548-istiod-ca-cert" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:07.155777 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155742    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-credential-socket" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:07.155938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155801    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-socket" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:07.155938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.155377    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-data" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:07.156081 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.156006    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.156081 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.156027    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.156081 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.156042    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/eed52cc6-2abf-47e5-979c-feadab32d548-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.156081 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.156059    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.156081 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.156075    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.157843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.157812    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-istio-token" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:07.157975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.157854    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/eed52cc6-2abf-47e5-979c-feadab32d548-istio-podinfo" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:10:07.157975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.157872    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-kube-api-access-gpfg2" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "kube-api-access-gpfg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:07.158409 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.158390    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-envoy" pod "eed52cc6-2abf-47e5-979c-feadab32d548" (UID: "eed52cc6-2abf-47e5-979c-feadab32d548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:10:07.257422 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.257334    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.257422 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.257373    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/eed52cc6-2abf-47e5-979c-feadab32d548-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.257422 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.257387    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpfg2\" (UniqueName: \"kubernetes.io/projected/eed52cc6-2abf-47e5-979c-feadab32d548-kube-api-access-gpfg2\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.257422 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.257401    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/eed52cc6-2abf-47e5-979c-feadab32d548-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:07.424493 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.424454    4291 generic.go:356] "Generic (PLEG): container finished" podID="eed52cc6-2abf-47e5-979c-feadab32d548" containerID="71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7" exitCode=0
Oct 30 14:10:07.424692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.424524    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"
Oct 30 14:10:07.424692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.424537    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" event={"ID":"eed52cc6-2abf-47e5-979c-feadab32d548","Type":"ContainerDied","Data":"71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7"}
Oct 30 14:10:07.424692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.424672    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8" event={"ID":"eed52cc6-2abf-47e5-979c-feadab32d548","Type":"ContainerDied","Data":"4403007b9404f0c82754877687503f1193a381ea6701ba13416722bf47a4e868"}
Oct 30 14:10:07.424895 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.424706    4291 scope.go:122] "RemoveContainer" containerID="71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7"
Oct 30 14:10:07.436350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.436330    4291 scope.go:122] "RemoveContainer" containerID="71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7"
Oct 30 14:10:07.436702 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:10:07.436679    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7\": container with ID starting with 71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7 not found: ID does not exist" containerID="71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7"
Oct 30 14:10:07.436788 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.436712    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7"} err="failed to get container status \"71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7\": rpc error: code = NotFound desc = could not find container \"71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7\": container with ID starting with 71c434bc2d4d32c75d0e2a91361e7953669d8e49fa7aa8a0ceaf96e059bcfee7 not found: ID does not exist"
Oct 30 14:10:07.476955 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.476918    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"]
Oct 30 14:10:07.480696 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.480669    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-9bf8ad-openshift-default-cb57f9b65-qf4q8"]
Oct 30 14:10:07.878948 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:07.878914    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" path="/var/lib/kubelet/pods/eed52cc6-2abf-47e5-979c-feadab32d548/volumes"
Oct 30 14:10:08.211518 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:10:08.211433    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:10:14.607319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:14.607282    4291 scope.go:122] "RemoveContainer" containerID="cce8e33f6f2a774a7e4c970a7a41a7660943c557881c6a44a5a89d416cfcc038"
Oct 30 14:10:18.931495 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.931450    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"]
Oct 30 14:10:18.932602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.932572    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7bf90d54-de4e-40f3-8995-08c2610e6172" containerName="revoke-keys"
Oct 30 14:10:18.932818 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.932801    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bf90d54-de4e-40f3-8995-08c2610e6172" containerName="revoke-keys"
Oct 30 14:10:18.933021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.933009    4291 cpu_manager.go:414] "containerMap: removing container" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy"
Oct 30 14:10:18.933105 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.933095    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy"
Oct 30 14:10:18.933484 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.933456    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="eed52cc6-2abf-47e5-979c-feadab32d548" containerName="istio-proxy"
Oct 30 14:10:18.933600 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.933495    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bf90d54-de4e-40f3-8995-08c2610e6172" containerName="revoke-keys"
Oct 30 14:10:18.938039 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.938011    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:18.950451 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.950422    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"]
Oct 30 14:10:18.952958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.952937    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:10:18.953802 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.953775    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:18.958083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.958061    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:10:18.958225 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.958115    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:10:18.994611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.994579    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:18.994779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.994726    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:18.994779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:18.994767    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wgpq\" (UniqueName: \"kubernetes.io/projected/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-kube-api-access-9wgpq\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.096305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.096250    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.096513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.096369    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.096513 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.096423    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wgpq\" (UniqueName: \"kubernetes.io/projected/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-kube-api-access-9wgpq\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.097121 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.097095    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.098827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.098806    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.116179 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.116154    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wgpq\" (UniqueName: \"kubernetes.io/projected/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-kube-api-access-9wgpq\") pod \"maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.254419 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.254337    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:19.556261 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:19.556230    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"]
Oct 30 14:10:19.557520 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:10:19.557491    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fe474d_26d3_4821_8d3e_53d9ddd2f79f.slice/crio-b8040d3167a3ae3e28c78b1804f19338e7067a2692da5f6ceea1244214eea499 WatchSource:0}: Error finding container b8040d3167a3ae3e28c78b1804f19338e7067a2692da5f6ceea1244214eea499: Status 404 returned error can't find the container with id b8040d3167a3ae3e28c78b1804f19338e7067a2692da5f6ceea1244214eea499
Oct 30 14:10:20.513572 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:20.513537    4291 generic.go:356] "Generic (PLEG): container finished" podID="08fe474d-26d3-4821-8d3e-53d9ddd2f79f" containerID="6863819d5fb5e807c37aed57bdd27f11f73eef778f0ab38a3584c139df710be4" exitCode=0
Oct 30 14:10:20.514004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:20.513591    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf" event={"ID":"08fe474d-26d3-4821-8d3e-53d9ddd2f79f","Type":"ContainerDied","Data":"6863819d5fb5e807c37aed57bdd27f11f73eef778f0ab38a3584c139df710be4"}
Oct 30 14:10:20.514004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:20.513663    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf" event={"ID":"08fe474d-26d3-4821-8d3e-53d9ddd2f79f","Type":"ContainerStarted","Data":"b8040d3167a3ae3e28c78b1804f19338e7067a2692da5f6ceea1244214eea499"}
Oct 30 14:10:21.666873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.666850    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:21.727330 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.727292    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-openshift-ca\") pod \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") "
Oct 30 14:10:21.727536 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.727372    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-xks-ca\") pod \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") "
Oct 30 14:10:21.727536 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.727481    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-kube-api-access-9wgpq\" (UniqueName: \"kubernetes.io/projected/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-kube-api-access-9wgpq\") pod \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\" (UID: \"08fe474d-26d3-4821-8d3e-53d9ddd2f79f\") "
Oct 30 14:10:21.727786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.727759    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-openshift-ca" pod "08fe474d-26d3-4821-8d3e-53d9ddd2f79f" (UID: "08fe474d-26d3-4821-8d3e-53d9ddd2f79f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:21.728158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.728133    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:21.730075 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.730042    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-kube-api-access-9wgpq" pod "08fe474d-26d3-4821-8d3e-53d9ddd2f79f" (UID: "08fe474d-26d3-4821-8d3e-53d9ddd2f79f"). InnerVolumeSpecName "kube-api-access-9wgpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:21.730225 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.730198    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-xks-ca" pod "08fe474d-26d3-4821-8d3e-53d9ddd2f79f" (UID: "08fe474d-26d3-4821-8d3e-53d9ddd2f79f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:21.829101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.829033    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:21.829101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:21.829060    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wgpq\" (UniqueName: \"kubernetes.io/projected/08fe474d-26d3-4821-8d3e-53d9ddd2f79f-kube-api-access-9wgpq\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:22.531107 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:22.531080    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"
Oct 30 14:10:22.531331 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:22.531130    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf" event={"ID":"08fe474d-26d3-4821-8d3e-53d9ddd2f79f","Type":"ContainerDied","Data":"b8040d3167a3ae3e28c78b1804f19338e7067a2692da5f6ceea1244214eea499"}
Oct 30 14:10:22.531331 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:22.531271    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8040d3167a3ae3e28c78b1804f19338e7067a2692da5f6ceea1244214eea499"
Oct 30 14:10:27.328771 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:27.328731    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:10:29.337249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:29.337212    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"]
Oct 30 14:10:33.664884 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.664844    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"]
Oct 30 14:10:33.665437 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.665422    4291 cpu_manager.go:414] "containerMap: removing container" podUID="08fe474d-26d3-4821-8d3e-53d9ddd2f79f" containerName="revoke-keys"
Oct 30 14:10:33.665484 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.665438    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08fe474d-26d3-4821-8d3e-53d9ddd2f79f" containerName="revoke-keys"
Oct 30 14:10:33.665579 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.665568    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="08fe474d-26d3-4821-8d3e-53d9ddd2f79f" containerName="revoke-keys"
Oct 30 14:10:33.669557 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.669534    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.686582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.686528    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"]
Oct 30 14:10:33.694696 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.694610    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:33.694696 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.694618    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:10:33.695860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.695836    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:10:33.695860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.695855    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:10:33.837784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.837724    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.838005 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.837923    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm99b\" (UniqueName: \"kubernetes.io/projected/28952843-8735-4ec2-8f0e-3f0186bf503a-kube-api-access-hm99b\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.838005 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.837968    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.939602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.939514    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.939792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.939739    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.939884 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.939871    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm99b\" (UniqueName: \"kubernetes.io/projected/28952843-8735-4ec2-8f0e-3f0186bf503a-kube-api-access-hm99b\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.940358 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.940325    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.942685 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.942656    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.959026 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.958989    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm99b\" (UniqueName: \"kubernetes.io/projected/28952843-8735-4ec2-8f0e-3f0186bf503a-kube-api-access-hm99b\") pod \"maas-api-revoke-sub-da174186-qg2d8\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:33.983843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:33.983805    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:34.288907 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:34.288817    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"]
Oct 30 14:10:34.338244 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:34.338203    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" podUID="1ef8317a-7398-4a40-860e-a8f15026eaef" containerName="payload-pre-processing" containerID="cri-o://ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9" gracePeriod=25
Oct 30 14:10:34.658046 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:34.658009    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8" event={"ID":"28952843-8735-4ec2-8f0e-3f0186bf503a","Type":"ContainerStarted","Data":"ec133b904d81d6e099cbee37afb313c9d8baad4c1c6d8f977c8ffd45d755ad6d"}
Oct 30 14:10:34.658250 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:34.658072    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8" event={"ID":"28952843-8735-4ec2-8f0e-3f0186bf503a","Type":"ContainerStarted","Data":"944b844e07b20043ccf252e5822c299e250802a2e7d68f88bf04977a7b708dcd"}
Oct 30 14:10:34.710419 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:34.710353    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8" podStartSLOduration=1.7103392849999999 podStartE2EDuration="1.710339285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:34.696595931 +0000 UTC m=+4044.956834166" watchObservedRunningTime="2026-10-30 14:10:34.710339285 +0000 UTC m=+4044.970577518"
Oct 30 14:10:35.666410 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:35.666375    4291 generic.go:356] "Generic (PLEG): container finished" podID="28952843-8735-4ec2-8f0e-3f0186bf503a" containerID="ec133b904d81d6e099cbee37afb313c9d8baad4c1c6d8f977c8ffd45d755ad6d" exitCode=0
Oct 30 14:10:35.666602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:35.666464    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8" event={"ID":"28952843-8735-4ec2-8f0e-3f0186bf503a","Type":"ContainerDied","Data":"ec133b904d81d6e099cbee37afb313c9d8baad4c1c6d8f977c8ffd45d755ad6d"}
Oct 30 14:10:36.658975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.658940    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"]
Oct 30 14:10:36.662972 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.662953    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.704716 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.704678    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"]
Oct 30 14:10:36.776498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.776464    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22fwg\" (UniqueName: \"kubernetes.io/projected/09c8df66-a002-4d78-8542-93d09b0a145d-kube-api-access-22fwg\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.776724 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.776630    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.776724 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.776713    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.839541 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.839511    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:36.879596 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.879036    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-xks-ca\") pod \"28952843-8735-4ec2-8f0e-3f0186bf503a\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") "
Oct 30 14:10:36.879596 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.879129    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28952843-8735-4ec2-8f0e-3f0186bf503a-kube-api-access-hm99b\" (UniqueName: \"kubernetes.io/projected/28952843-8735-4ec2-8f0e-3f0186bf503a-kube-api-access-hm99b\") pod \"28952843-8735-4ec2-8f0e-3f0186bf503a\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") "
Oct 30 14:10:36.879596 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.879541    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-openshift-ca\") pod \"28952843-8735-4ec2-8f0e-3f0186bf503a\" (UID: \"28952843-8735-4ec2-8f0e-3f0186bf503a\") "
Oct 30 14:10:36.880224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.879804    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.880224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.879871    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.880224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.880037    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22fwg\" (UniqueName: \"kubernetes.io/projected/09c8df66-a002-4d78-8542-93d09b0a145d-kube-api-access-22fwg\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.881173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.881144    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-xks-ca" pod "28952843-8735-4ec2-8f0e-3f0186bf503a" (UID: "28952843-8735-4ec2-8f0e-3f0186bf503a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:36.881664 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.881610    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-openshift-ca" pod "28952843-8735-4ec2-8f0e-3f0186bf503a" (UID: "28952843-8735-4ec2-8f0e-3f0186bf503a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:36.881913 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.881880    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.884222 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.884193    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.887780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.887756    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28952843-8735-4ec2-8f0e-3f0186bf503a-kube-api-access-hm99b" pod "28952843-8735-4ec2-8f0e-3f0186bf503a" (UID: "28952843-8735-4ec2-8f0e-3f0186bf503a"). InnerVolumeSpecName "kube-api-access-hm99b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:36.899481 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.899449    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-22fwg\" (UniqueName: \"kubernetes.io/projected/09c8df66-a002-4d78-8542-93d09b0a145d-kube-api-access-22fwg\") pod \"maas-api-revoke-sub-741d51c5-p7zg6\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.978095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.978004    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:36.981788 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.981760    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:36.981890 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.981792    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28952843-8735-4ec2-8f0e-3f0186bf503a-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:36.981890 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:36.981806    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm99b\" (UniqueName: \"kubernetes.io/projected/28952843-8735-4ec2-8f0e-3f0186bf503a-kube-api-access-hm99b\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:37.287061 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:37.287028    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"]
Oct 30 14:10:37.288610 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:10:37.288586    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09c8df66_a002_4d78_8542_93d09b0a145d.slice/crio-44b32dad0952780bd08c747ed852e8f5cee9d1ea850ba18a2bc8fcebdb40f5e8 WatchSource:0}: Error finding container 44b32dad0952780bd08c747ed852e8f5cee9d1ea850ba18a2bc8fcebdb40f5e8: Status 404 returned error can't find the container with id 44b32dad0952780bd08c747ed852e8f5cee9d1ea850ba18a2bc8fcebdb40f5e8
Oct 30 14:10:37.682442 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:37.682402    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6" event={"ID":"09c8df66-a002-4d78-8542-93d09b0a145d","Type":"ContainerStarted","Data":"e09e5d89d6a933c8ac595aff5c8a100ace41b8dc8fbf5f3ee7039165fedd33ba"}
Oct 30 14:10:37.682988 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:37.682462    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6" event={"ID":"09c8df66-a002-4d78-8542-93d09b0a145d","Type":"ContainerStarted","Data":"44b32dad0952780bd08c747ed852e8f5cee9d1ea850ba18a2bc8fcebdb40f5e8"}
Oct 30 14:10:37.688453 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:37.688428    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8"
Oct 30 14:10:37.688665 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:37.688465    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da174186-qg2d8" event={"ID":"28952843-8735-4ec2-8f0e-3f0186bf503a","Type":"ContainerDied","Data":"944b844e07b20043ccf252e5822c299e250802a2e7d68f88bf04977a7b708dcd"}
Oct 30 14:10:37.688665 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:37.688510    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="944b844e07b20043ccf252e5822c299e250802a2e7d68f88bf04977a7b708dcd"
Oct 30 14:10:37.726974 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:37.726903    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6" podStartSLOduration=1.726883156 podStartE2EDuration="1.726883156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:37.720070258 +0000 UTC m=+4047.980308492" watchObservedRunningTime="2026-10-30 14:10:37.726883156 +0000 UTC m=+4047.987121392"
Oct 30 14:10:38.210923 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:10:38.210882    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:10:38.696830 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:38.696787    4291 generic.go:356] "Generic (PLEG): container finished" podID="09c8df66-a002-4d78-8542-93d09b0a145d" containerID="e09e5d89d6a933c8ac595aff5c8a100ace41b8dc8fbf5f3ee7039165fedd33ba" exitCode=0
Oct 30 14:10:38.697268 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:38.696872    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6" event={"ID":"09c8df66-a002-4d78-8542-93d09b0a145d","Type":"ContainerDied","Data":"e09e5d89d6a933c8ac595aff5c8a100ace41b8dc8fbf5f3ee7039165fedd33ba"}
Oct 30 14:10:39.851757 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.851732    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:39.919153 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.919122    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-xks-ca\") pod \"09c8df66-a002-4d78-8542-93d09b0a145d\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") "
Oct 30 14:10:39.919357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.919184    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-openshift-ca\") pod \"09c8df66-a002-4d78-8542-93d09b0a145d\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") "
Oct 30 14:10:39.919357 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.919243    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09c8df66-a002-4d78-8542-93d09b0a145d-kube-api-access-22fwg\" (UniqueName: \"kubernetes.io/projected/09c8df66-a002-4d78-8542-93d09b0a145d-kube-api-access-22fwg\") pod \"09c8df66-a002-4d78-8542-93d09b0a145d\" (UID: \"09c8df66-a002-4d78-8542-93d09b0a145d\") "
Oct 30 14:10:39.919613 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.919577    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-openshift-ca" pod "09c8df66-a002-4d78-8542-93d09b0a145d" (UID: "09c8df66-a002-4d78-8542-93d09b0a145d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:39.920004 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.919978    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:39.921561 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.921535    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09c8df66-a002-4d78-8542-93d09b0a145d-kube-api-access-22fwg" pod "09c8df66-a002-4d78-8542-93d09b0a145d" (UID: "09c8df66-a002-4d78-8542-93d09b0a145d"). InnerVolumeSpecName "kube-api-access-22fwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:39.921727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:39.921711    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-xks-ca" pod "09c8df66-a002-4d78-8542-93d09b0a145d" (UID: "09c8df66-a002-4d78-8542-93d09b0a145d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:40.021722 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:40.021602    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09c8df66-a002-4d78-8542-93d09b0a145d-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.021722 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:40.021667    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22fwg\" (UniqueName: \"kubernetes.io/projected/09c8df66-a002-4d78-8542-93d09b0a145d-kube-api-access-22fwg\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:40.711703 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:40.711666    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6" event={"ID":"09c8df66-a002-4d78-8542-93d09b0a145d","Type":"ContainerDied","Data":"44b32dad0952780bd08c747ed852e8f5cee9d1ea850ba18a2bc8fcebdb40f5e8"}
Oct 30 14:10:40.711893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:40.711726    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="44b32dad0952780bd08c747ed852e8f5cee9d1ea850ba18a2bc8fcebdb40f5e8"
Oct 30 14:10:40.711893 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:40.711677    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-741d51c5-p7zg6"
Oct 30 14:10:40.808351 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:40.808203    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"]
Oct 30 14:10:40.812679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:40.812622    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-74589c-s84qf"]
Oct 30 14:10:41.879610 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:41.879574    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08fe474d-26d3-4821-8d3e-53d9ddd2f79f" path="/var/lib/kubelet/pods/08fe474d-26d3-4821-8d3e-53d9ddd2f79f/volumes"
Oct 30 14:10:43.926228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.926186    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" podUID="1ef8317a-7398-4a40-860e-a8f15026eaef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:10:43.973918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.973871    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"]
Oct 30 14:10:43.974696 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.974676    4291 cpu_manager.go:414] "containerMap: removing container" podUID="09c8df66-a002-4d78-8542-93d09b0a145d" containerName="revoke-keys"
Oct 30 14:10:43.974757 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.974700    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09c8df66-a002-4d78-8542-93d09b0a145d" containerName="revoke-keys"
Oct 30 14:10:43.974757 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.974733    4291 cpu_manager.go:414] "containerMap: removing container" podUID="28952843-8735-4ec2-8f0e-3f0186bf503a" containerName="revoke-keys"
Oct 30 14:10:43.974757 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.974743    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28952843-8735-4ec2-8f0e-3f0186bf503a" containerName="revoke-keys"
Oct 30 14:10:43.974861 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.974844    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="28952843-8735-4ec2-8f0e-3f0186bf503a" containerName="revoke-keys"
Oct 30 14:10:43.974896 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.974863    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="09c8df66-a002-4d78-8542-93d09b0a145d" containerName="revoke-keys"
Oct 30 14:10:43.978497 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.978479    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:43.995231 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:43.995191    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"]
Oct 30 14:10:44.001905 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.001875    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:10:44.002057 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.001941    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:10:44.005058 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.005029    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:44.005192 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.005069    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:10:44.076224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.076187    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhtn9\" (UniqueName: \"kubernetes.io/projected/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-kube-api-access-nhtn9\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.076224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.076230    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.076508 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.076465    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.177686 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.177550    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.177937 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.177710    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhtn9\" (UniqueName: \"kubernetes.io/projected/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-kube-api-access-nhtn9\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.177937 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.177744    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.178248 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.178225    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.180249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.180225    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.197141 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.197104    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhtn9\" (UniqueName: \"kubernetes.io/projected/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-kube-api-access-nhtn9\") pod \"maas-api-revoke-sub-e8c6d14a-vs4gq\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.292461 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.292426    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:44.626794 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.626760    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"]
Oct 30 14:10:44.628489 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:10:44.628459    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca90cf69_045e_4f21_b8aa_9781ebb7d67e.slice/crio-c6d111fbf74cb0785005d30d5dfb0fa328c27904cc8415ca7efe5ec05610603a WatchSource:0}: Error finding container c6d111fbf74cb0785005d30d5dfb0fa328c27904cc8415ca7efe5ec05610603a: Status 404 returned error can't find the container with id c6d111fbf74cb0785005d30d5dfb0fa328c27904cc8415ca7efe5ec05610603a
Oct 30 14:10:44.741270 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:44.741231    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq" event={"ID":"ca90cf69-045e-4f21-b8aa-9781ebb7d67e","Type":"ContainerStarted","Data":"c6d111fbf74cb0785005d30d5dfb0fa328c27904cc8415ca7efe5ec05610603a"}
Oct 30 14:10:45.748885 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:45.748849    4291 generic.go:356] "Generic (PLEG): container finished" podID="ca90cf69-045e-4f21-b8aa-9781ebb7d67e" containerID="05e8c10dd9b24890e408165c9a89c8e067bf29ae5d59b9ec918f99b23699ef7a" exitCode=0
Oct 30 14:10:45.749403 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:45.748938    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq" event={"ID":"ca90cf69-045e-4f21-b8aa-9781ebb7d67e","Type":"ContainerDied","Data":"05e8c10dd9b24890e408165c9a89c8e067bf29ae5d59b9ec918f99b23699ef7a"}
Oct 30 14:10:46.921395 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:46.921366    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:47.001058 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.001022    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"]
Oct 30 14:10:47.002104 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.002071    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ca90cf69-045e-4f21-b8aa-9781ebb7d67e" containerName="revoke-keys"
Oct 30 14:10:47.002104 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.002104    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca90cf69-045e-4f21-b8aa-9781ebb7d67e" containerName="revoke-keys"
Oct 30 14:10:47.002332 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.002318    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca90cf69-045e-4f21-b8aa-9781ebb7d67e" containerName="revoke-keys"
Oct 30 14:10:47.006455 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.006386    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.013986 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.013955    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-xks-ca\") pod \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") "
Oct 30 14:10:47.014146 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.014076    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-kube-api-access-nhtn9\" (UniqueName: \"kubernetes.io/projected/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-kube-api-access-nhtn9\") pod \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") "
Oct 30 14:10:47.014359 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.014148    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-openshift-ca\") pod \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\" (UID: \"ca90cf69-045e-4f21-b8aa-9781ebb7d67e\") "
Oct 30 14:10:47.015043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.015010    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-openshift-ca" pod "ca90cf69-045e-4f21-b8aa-9781ebb7d67e" (UID: "ca90cf69-045e-4f21-b8aa-9781ebb7d67e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:47.017281 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.017252    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-xks-ca" pod "ca90cf69-045e-4f21-b8aa-9781ebb7d67e" (UID: "ca90cf69-045e-4f21-b8aa-9781ebb7d67e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:47.024157 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.024128    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"]
Oct 30 14:10:47.026847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.026818    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-kube-api-access-nhtn9" pod "ca90cf69-045e-4f21-b8aa-9781ebb7d67e" (UID: "ca90cf69-045e-4f21-b8aa-9781ebb7d67e"). InnerVolumeSpecName "kube-api-access-nhtn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:47.116043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.116004    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.116230 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.116118    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.116280 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.116236    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6f9n\" (UniqueName: \"kubernetes.io/projected/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-kube-api-access-p6f9n\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.116368 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.116354    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:47.116416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.116374    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhtn9\" (UniqueName: \"kubernetes.io/projected/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-kube-api-access-nhtn9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:47.116416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.116390    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca90cf69-045e-4f21-b8aa-9781ebb7d67e-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:47.217179 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.217138    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.217405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.217219    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.217405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.217343    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6f9n\" (UniqueName: \"kubernetes.io/projected/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-kube-api-access-p6f9n\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.217974 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.217946    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.219804 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.219781    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.236940 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.236909    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6f9n\" (UniqueName: \"kubernetes.io/projected/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-kube-api-access-p6f9n\") pod \"maas-api-revoke-sub-32a081c7-d4pvp\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.331174 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.331083    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:47.645212 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.645167    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"]
Oct 30 14:10:47.763129 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.763102    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq"
Oct 30 14:10:47.763338 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.763141    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8c6d14a-vs4gq" event={"ID":"ca90cf69-045e-4f21-b8aa-9781ebb7d67e","Type":"ContainerDied","Data":"c6d111fbf74cb0785005d30d5dfb0fa328c27904cc8415ca7efe5ec05610603a"}
Oct 30 14:10:47.763338 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.763194    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6d111fbf74cb0785005d30d5dfb0fa328c27904cc8415ca7efe5ec05610603a"
Oct 30 14:10:47.764691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:47.764664    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp" event={"ID":"f9fe5005-7453-4b5e-a3c7-d6d866a42de3","Type":"ContainerStarted","Data":"d2e55d25922947ccd84cf521049f29c679ef93a55d7a35faf276a3b2ce563d5b"}
Oct 30 14:10:48.773512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:48.773471    4291 generic.go:356] "Generic (PLEG): container finished" podID="f9fe5005-7453-4b5e-a3c7-d6d866a42de3" containerID="ac165d5c63a3ba51eddc9a6e4d6d2801bbe90103f2d924b4ef54f153c6c027ec" exitCode=0
Oct 30 14:10:48.774011 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:48.773557    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp" event={"ID":"f9fe5005-7453-4b5e-a3c7-d6d866a42de3","Type":"ContainerDied","Data":"ac165d5c63a3ba51eddc9a6e4d6d2801bbe90103f2d924b4ef54f153c6c027ec"}
Oct 30 14:10:49.951674 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:49.951611    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:50.076847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.076754    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-kube-api-access-p6f9n\" (UniqueName: \"kubernetes.io/projected/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-kube-api-access-p6f9n\") pod \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") "
Oct 30 14:10:50.076847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.076829    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-openshift-ca\") pod \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") "
Oct 30 14:10:50.077102 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.076938    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-xks-ca\") pod \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\" (UID: \"f9fe5005-7453-4b5e-a3c7-d6d866a42de3\") "
Oct 30 14:10:50.077225 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.077199    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-openshift-ca" pod "f9fe5005-7453-4b5e-a3c7-d6d866a42de3" (UID: "f9fe5005-7453-4b5e-a3c7-d6d866a42de3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:50.077323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.077307    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:50.079144 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.079118    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-xks-ca" pod "f9fe5005-7453-4b5e-a3c7-d6d866a42de3" (UID: "f9fe5005-7453-4b5e-a3c7-d6d866a42de3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:50.079313 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.079292    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-kube-api-access-p6f9n" pod "f9fe5005-7453-4b5e-a3c7-d6d866a42de3" (UID: "f9fe5005-7453-4b5e-a3c7-d6d866a42de3"). InnerVolumeSpecName "kube-api-access-p6f9n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:50.172387 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.172344    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"]
Oct 30 14:10:50.173211 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.173184    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f9fe5005-7453-4b5e-a3c7-d6d866a42de3" containerName="revoke-keys"
Oct 30 14:10:50.173323 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.173214    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9fe5005-7453-4b5e-a3c7-d6d866a42de3" containerName="revoke-keys"
Oct 30 14:10:50.173412 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.173397    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9fe5005-7453-4b5e-a3c7-d6d866a42de3" containerName="revoke-keys"
Oct 30 14:10:50.177355 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.177324    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.178414 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.178379    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:50.178509 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.178420    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6f9n\" (UniqueName: \"kubernetes.io/projected/f9fe5005-7453-4b5e-a3c7-d6d866a42de3-kube-api-access-p6f9n\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:50.193782 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.193743    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"]
Oct 30 14:10:50.279886 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.279843    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.280079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.280009    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.280079 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.280044    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvfq\" (UniqueName: \"kubernetes.io/projected/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-kube-api-access-rqvfq\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.381094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.381054    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.381273 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.381154    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.381273 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.381177    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvfq\" (UniqueName: \"kubernetes.io/projected/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-kube-api-access-rqvfq\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.381958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.381936    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.383716 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.383693    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.407971 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.407935    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvfq\" (UniqueName: \"kubernetes.io/projected/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-kube-api-access-rqvfq\") pod \"maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.493426 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.493388    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:50.790854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.790772    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp"
Oct 30 14:10:50.790854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.790806    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32a081c7-d4pvp" event={"ID":"f9fe5005-7453-4b5e-a3c7-d6d866a42de3","Type":"ContainerDied","Data":"d2e55d25922947ccd84cf521049f29c679ef93a55d7a35faf276a3b2ce563d5b"}
Oct 30 14:10:50.790854 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.790852    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2e55d25922947ccd84cf521049f29c679ef93a55d7a35faf276a3b2ce563d5b"
Oct 30 14:10:50.799232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:50.799202    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"]
Oct 30 14:10:50.802657 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:10:50.802611    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ddf1edb_e4c4_45e0_8607_3c19f4076ec0.slice/crio-d255b76c3a6e344688ba72beeac55e8947b45d18dff0ec26b63a78c3969eeab7 WatchSource:0}: Error finding container d255b76c3a6e344688ba72beeac55e8947b45d18dff0ec26b63a78c3969eeab7: Status 404 returned error can't find the container with id d255b76c3a6e344688ba72beeac55e8947b45d18dff0ec26b63a78c3969eeab7
Oct 30 14:10:51.800043 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:51.799997    4291 generic.go:356] "Generic (PLEG): container finished" podID="8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" containerID="cc8b87ae80085cc9d9a13740e5725b56b9cdfc7abe6c299c37b9b355a0b8dded" exitCode=0
Oct 30 14:10:51.800458 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:51.800087    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs" event={"ID":"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0","Type":"ContainerDied","Data":"cc8b87ae80085cc9d9a13740e5725b56b9cdfc7abe6c299c37b9b355a0b8dded"}
Oct 30 14:10:51.800458 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:51.800134    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs" event={"ID":"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0","Type":"ContainerStarted","Data":"d255b76c3a6e344688ba72beeac55e8947b45d18dff0ec26b63a78c3969eeab7"}
Oct 30 14:10:52.988505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:52.988476    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:53.013003 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.012968    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-openshift-ca\") pod \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") "
Oct 30 14:10:53.013176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.013040    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-kube-api-access-rqvfq\" (UniqueName: \"kubernetes.io/projected/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-kube-api-access-rqvfq\") pod \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") "
Oct 30 14:10:53.013176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.013108    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-xks-ca\") pod \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\" (UID: \"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0\") "
Oct 30 14:10:53.013367 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.013344    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-openshift-ca" pod "8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" (UID: "8ddf1edb-e4c4-45e0-8607-3c19f4076ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:53.013595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.013571    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:53.015375 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.015347    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-xks-ca" pod "8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" (UID: "8ddf1edb-e4c4-45e0-8607-3c19f4076ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:10:53.015375 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.015370    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-kube-api-access-rqvfq" pod "8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" (UID: "8ddf1edb-e4c4-45e0-8607-3c19f4076ec0"). InnerVolumeSpecName "kube-api-access-rqvfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:53.115071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.115037    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqvfq\" (UniqueName: \"kubernetes.io/projected/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-kube-api-access-rqvfq\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:53.115071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.115070    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:53.814292 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.814265    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"
Oct 30 14:10:53.814477 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.814259    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs" event={"ID":"8ddf1edb-e4c4-45e0-8607-3c19f4076ec0","Type":"ContainerDied","Data":"d255b76c3a6e344688ba72beeac55e8947b45d18dff0ec26b63a78c3969eeab7"}
Oct 30 14:10:53.814477 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.814387    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d255b76c3a6e344688ba72beeac55e8947b45d18dff0ec26b63a78c3969eeab7"
Oct 30 14:10:53.925727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:53.925688    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" podUID="1ef8317a-7398-4a40-860e-a8f15026eaef" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:10:54.595719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.595689    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 14:10:54.631374 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.631328    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ef8317a-7398-4a40-860e-a8f15026eaef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef8317a-7398-4a40-860e-a8f15026eaef-plugins-config-volume\") pod \"1ef8317a-7398-4a40-860e-a8f15026eaef\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") "
Oct 30 14:10:54.631538 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.631500    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ef8317a-7398-4a40-860e-a8f15026eaef-kube-api-access-5w86x\" (UniqueName: \"kubernetes.io/projected/1ef8317a-7398-4a40-860e-a8f15026eaef-kube-api-access-5w86x\") pod \"1ef8317a-7398-4a40-860e-a8f15026eaef\" (UID: \"1ef8317a-7398-4a40-860e-a8f15026eaef\") "
Oct 30 14:10:54.631847 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.631821    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ef8317a-7398-4a40-860e-a8f15026eaef-plugins-config-volume" pod "1ef8317a-7398-4a40-860e-a8f15026eaef" (UID: "1ef8317a-7398-4a40-860e-a8f15026eaef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:10:54.632100 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.632078    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ef8317a-7398-4a40-860e-a8f15026eaef-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:54.633608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.633584    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ef8317a-7398-4a40-860e-a8f15026eaef-kube-api-access-5w86x" pod "1ef8317a-7398-4a40-860e-a8f15026eaef" (UID: "1ef8317a-7398-4a40-860e-a8f15026eaef"). InnerVolumeSpecName "kube-api-access-5w86x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:10:54.733557 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.733522    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5w86x\" (UniqueName: \"kubernetes.io/projected/1ef8317a-7398-4a40-860e-a8f15026eaef-kube-api-access-5w86x\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:10:54.823093 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.823057    4291 generic.go:356] "Generic (PLEG): container finished" podID="1ef8317a-7398-4a40-860e-a8f15026eaef" containerID="ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9" exitCode=0
Oct 30 14:10:54.823276 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.823112    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"
Oct 30 14:10:54.823276 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.823139    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" event={"ID":"1ef8317a-7398-4a40-860e-a8f15026eaef","Type":"ContainerDied","Data":"ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9"}
Oct 30 14:10:54.823276 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.823199    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk" event={"ID":"1ef8317a-7398-4a40-860e-a8f15026eaef","Type":"ContainerDied","Data":"90853881f602c0af98678460019c04aad9b95630d8597f2c84ba65ce5f81f5e1"}
Oct 30 14:10:54.823276 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.823224    4291 scope.go:122] "RemoveContainer" containerID="ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9"
Oct 30 14:10:54.835880 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.835855    4291 scope.go:122] "RemoveContainer" containerID="ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9"
Oct 30 14:10:54.836342 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:10:54.836304    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9\": container with ID starting with ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9 not found: ID does not exist" containerID="ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9"
Oct 30 14:10:54.836498 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.836352    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9"} err="failed to get container status \"ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9\": rpc error: code = NotFound desc = could not find container \"ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9\": container with ID starting with ba0cd91980c595a282f83941931ac040444f557cce37df8c01e490c483c0faa9 not found: ID does not exist"
Oct 30 14:10:54.874824 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.874736    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"]
Oct 30 14:10:54.881219 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:54.881189    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-74589c-5554f8475f-lwbvk"]
Oct 30 14:10:55.878289 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:55.878248    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ef8317a-7398-4a40-860e-a8f15026eaef" path="/var/lib/kubelet/pods/1ef8317a-7398-4a40-860e-a8f15026eaef/volumes"
Oct 30 14:10:56.792650 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.792613    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"]
Oct 30 14:10:56.793253 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.793236    4291 cpu_manager.go:414] "containerMap: removing container" podUID="1ef8317a-7398-4a40-860e-a8f15026eaef" containerName="payload-pre-processing"
Oct 30 14:10:56.793367 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.793253    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ef8317a-7398-4a40-860e-a8f15026eaef" containerName="payload-pre-processing"
Oct 30 14:10:56.793367 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.793276    4291 cpu_manager.go:414] "containerMap: removing container" podUID="8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" containerName="revoke-keys"
Oct 30 14:10:56.793367 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.793284    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" containerName="revoke-keys"
Oct 30 14:10:56.793548 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.793378    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" containerName="revoke-keys"
Oct 30 14:10:56.793548 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.793395    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ef8317a-7398-4a40-860e-a8f15026eaef" containerName="payload-pre-processing"
Oct 30 14:10:56.798470 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.798450    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.812861 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.812833    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"]
Oct 30 14:10:56.813123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.813073    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:10:56.813123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.813087    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:10:56.813123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.813111    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:10:56.820647 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.820614    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:10:56.878574 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.878540    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpw69\" (UniqueName: \"kubernetes.io/projected/d7d9f56a-ee59-492a-8f08-239985962d91-kube-api-access-bpw69\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.878969 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.878602    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.878969 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.878844    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.979995 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.979958    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpw69\" (UniqueName: \"kubernetes.io/projected/d7d9f56a-ee59-492a-8f08-239985962d91-kube-api-access-bpw69\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.980169 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.980142    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.980429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.980407    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.981025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.981007    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.982517 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.982496    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:56.998463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:56.998427    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpw69\" (UniqueName: \"kubernetes.io/projected/d7d9f56a-ee59-492a-8f08-239985962d91-kube-api-access-bpw69\") pod \"maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:57.113873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:57.113838    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:10:57.418843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:57.418813    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"]
Oct 30 14:10:57.419769 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:10:57.419742    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7d9f56a_ee59_492a_8f08_239985962d91.slice/crio-aa19ee04910c731ea1548574056d77a9c30186a1514e37c5f42b3b5f596eb080 WatchSource:0}: Error finding container aa19ee04910c731ea1548574056d77a9c30186a1514e37c5f42b3b5f596eb080: Status 404 returned error can't find the container with id aa19ee04910c731ea1548574056d77a9c30186a1514e37c5f42b3b5f596eb080
Oct 30 14:10:57.847127 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:57.847085    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz" event={"ID":"d7d9f56a-ee59-492a-8f08-239985962d91","Type":"ContainerStarted","Data":"8ceaeaf5a8263ed01ca0a9d295c2b1e867037ae6182bb687ec545ad557927e7c"}
Oct 30 14:10:57.847334 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:57.847148    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz" event={"ID":"d7d9f56a-ee59-492a-8f08-239985962d91","Type":"ContainerStarted","Data":"aa19ee04910c731ea1548574056d77a9c30186a1514e37c5f42b3b5f596eb080"}
Oct 30 14:10:57.885429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:57.885381    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz" podStartSLOduration=1.885368707 podStartE2EDuration="1.885368707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:10:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:10:57.883924744 +0000 UTC m=+4068.144162979" watchObservedRunningTime="2026-10-30 14:10:57.885368707 +0000 UTC m=+4068.145606941"
Oct 30 14:10:58.079096 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.079010    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"]
Oct 30 14:10:58.083027 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.083010    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.101677 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.101650    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"]
Oct 30 14:10:58.195548 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.195513    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46198189-2833-4782-9772-193ca26c986c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.195743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.195664    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46198189-2833-4782-9772-193ca26c986c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.195743 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.195710    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8s926\" (UniqueName: \"kubernetes.io/projected/46198189-2833-4782-9772-193ca26c986c-kube-api-access-8s926\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.297450 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.297422    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46198189-2833-4782-9772-193ca26c986c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.297555 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.297475    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8s926\" (UniqueName: \"kubernetes.io/projected/46198189-2833-4782-9772-193ca26c986c-kube-api-access-8s926\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.297695 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.297675    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46198189-2833-4782-9772-193ca26c986c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.298151 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.298129    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46198189-2833-4782-9772-193ca26c986c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.299792 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.299775    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46198189-2833-4782-9772-193ca26c986c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.316104 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.316077    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8s926\" (UniqueName: \"kubernetes.io/projected/46198189-2833-4782-9772-193ca26c986c-kube-api-access-8s926\") pod \"maas-api-revoke-sub-e1520bde-8jzhd\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.397915 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.397876    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:10:58.702465 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.702432    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"]
Oct 30 14:10:58.703508 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:10:58.703473    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46198189_2833_4782_9772_193ca26c986c.slice/crio-cff86d5157d3919e87bd335185955f4349db53daf1d08e4ddded83f6899428d3 WatchSource:0}: Error finding container cff86d5157d3919e87bd335185955f4349db53daf1d08e4ddded83f6899428d3: Status 404 returned error can't find the container with id cff86d5157d3919e87bd335185955f4349db53daf1d08e4ddded83f6899428d3
Oct 30 14:10:58.859009 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.858975    4291 generic.go:356] "Generic (PLEG): container finished" podID="d7d9f56a-ee59-492a-8f08-239985962d91" containerID="8ceaeaf5a8263ed01ca0a9d295c2b1e867037ae6182bb687ec545ad557927e7c" exitCode=0
Oct 30 14:10:58.859189 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.859021    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz" event={"ID":"d7d9f56a-ee59-492a-8f08-239985962d91","Type":"ContainerDied","Data":"8ceaeaf5a8263ed01ca0a9d295c2b1e867037ae6182bb687ec545ad557927e7c"}
Oct 30 14:10:58.860308 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:58.860283    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd" event={"ID":"46198189-2833-4782-9772-193ca26c986c","Type":"ContainerStarted","Data":"cff86d5157d3919e87bd335185955f4349db53daf1d08e4ddded83f6899428d3"}
Oct 30 14:10:59.868523 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:59.868485    4291 generic.go:356] "Generic (PLEG): container finished" podID="46198189-2833-4782-9772-193ca26c986c" containerID="fc7cddd041fee02eb4c6f6cc792643c22695dda0904e0e6e58efa5f30fa3293e" exitCode=0
Oct 30 14:10:59.869006 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:10:59.868570    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd" event={"ID":"46198189-2833-4782-9772-193ca26c986c","Type":"ContainerDied","Data":"fc7cddd041fee02eb4c6f6cc792643c22695dda0904e0e6e58efa5f30fa3293e"}
Oct 30 14:11:00.066514 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.066481    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:11:00.121905 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.121823    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d7d9f56a-ee59-492a-8f08-239985962d91-kube-api-access-bpw69\" (UniqueName: \"kubernetes.io/projected/d7d9f56a-ee59-492a-8f08-239985962d91-kube-api-access-bpw69\") pod \"d7d9f56a-ee59-492a-8f08-239985962d91\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") "
Oct 30 14:11:00.122101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.121927    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-xks-ca\") pod \"d7d9f56a-ee59-492a-8f08-239985962d91\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") "
Oct 30 14:11:00.122101 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.121957    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-openshift-ca\") pod \"d7d9f56a-ee59-492a-8f08-239985962d91\" (UID: \"d7d9f56a-ee59-492a-8f08-239985962d91\") "
Oct 30 14:11:00.122462 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.122429    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-openshift-ca" pod "d7d9f56a-ee59-492a-8f08-239985962d91" (UID: "d7d9f56a-ee59-492a-8f08-239985962d91"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:00.124139 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.124112    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d7d9f56a-ee59-492a-8f08-239985962d91-kube-api-access-bpw69" pod "d7d9f56a-ee59-492a-8f08-239985962d91" (UID: "d7d9f56a-ee59-492a-8f08-239985962d91"). InnerVolumeSpecName "kube-api-access-bpw69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:00.125570 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.125545    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-xks-ca" pod "d7d9f56a-ee59-492a-8f08-239985962d91" (UID: "d7d9f56a-ee59-492a-8f08-239985962d91"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:00.223602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.223563    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bpw69\" (UniqueName: \"kubernetes.io/projected/d7d9f56a-ee59-492a-8f08-239985962d91-kube-api-access-bpw69\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:00.223602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.223599    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:00.223867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.223613    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d7d9f56a-ee59-492a-8f08-239985962d91-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:00.686314 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.686277    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"]
Oct 30 14:11:00.715943 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.715912    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"]
Oct 30 14:11:00.877520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.877493    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"
Oct 30 14:11:00.877994 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.877525    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz" event={"ID":"d7d9f56a-ee59-492a-8f08-239985962d91","Type":"ContainerDied","Data":"aa19ee04910c731ea1548574056d77a9c30186a1514e37c5f42b3b5f596eb080"}
Oct 30 14:11:00.877994 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:00.877573    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa19ee04910c731ea1548574056d77a9c30186a1514e37c5f42b3b5f596eb080"
Oct 30 14:11:01.055902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.055880    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:11:01.137579 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.137548    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46198189-2833-4782-9772-193ca26c986c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46198189-2833-4782-9772-193ca26c986c-maas-api-openshift-ca\") pod \"46198189-2833-4782-9772-193ca26c986c\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") "
Oct 30 14:11:01.137772 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.137669    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46198189-2833-4782-9772-193ca26c986c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46198189-2833-4782-9772-193ca26c986c-maas-api-xks-ca\") pod \"46198189-2833-4782-9772-193ca26c986c\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") "
Oct 30 14:11:01.137772 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.137709    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46198189-2833-4782-9772-193ca26c986c-kube-api-access-8s926\" (UniqueName: \"kubernetes.io/projected/46198189-2833-4782-9772-193ca26c986c-kube-api-access-8s926\") pod \"46198189-2833-4782-9772-193ca26c986c\" (UID: \"46198189-2833-4782-9772-193ca26c986c\") "
Oct 30 14:11:01.137997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.137966    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46198189-2833-4782-9772-193ca26c986c-maas-api-openshift-ca" pod "46198189-2833-4782-9772-193ca26c986c" (UID: "46198189-2833-4782-9772-193ca26c986c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:01.138204 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.138184    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/46198189-2833-4782-9772-193ca26c986c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:01.139939 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.139915    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46198189-2833-4782-9772-193ca26c986c-maas-api-xks-ca" pod "46198189-2833-4782-9772-193ca26c986c" (UID: "46198189-2833-4782-9772-193ca26c986c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:01.140071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.139943    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46198189-2833-4782-9772-193ca26c986c-kube-api-access-8s926" pod "46198189-2833-4782-9772-193ca26c986c" (UID: "46198189-2833-4782-9772-193ca26c986c"). InnerVolumeSpecName "kube-api-access-8s926". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:01.259710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.244947    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/46198189-2833-4782-9772-193ca26c986c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:01.259710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.244990    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8s926\" (UniqueName: \"kubernetes.io/projected/46198189-2833-4782-9772-193ca26c986c-kube-api-access-8s926\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:01.893114 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.893090    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd"
Oct 30 14:11:01.893589 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.893077    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e1520bde-8jzhd" event={"ID":"46198189-2833-4782-9772-193ca26c986c","Type":"ContainerDied","Data":"cff86d5157d3919e87bd335185955f4349db53daf1d08e4ddded83f6899428d3"}
Oct 30 14:11:01.893589 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:01.893230    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff86d5157d3919e87bd335185955f4349db53daf1d08e4ddded83f6899428d3"
Oct 30 14:11:05.687149 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:05.687111    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" podUID="7fc89d43-2b87-4184-904e-8d86482b70a6" containerName="payload-processing" containerID="cri-o://8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c" gracePeriod=25
Oct 30 14:11:05.716995 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:05.716951    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" podUID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" containerName="payload-pre-processing" containerID="cri-o://1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d" gracePeriod=25
Oct 30 14:11:07.186246 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:07.186210    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"]
Oct 30 14:11:08.212507 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:08.212467    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:11:08.523271 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.523179    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"]
Oct 30 14:11:08.534597 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.534568    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-98f1cc-c7hxs"]
Oct 30 14:11:08.600169 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.600132    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"]
Oct 30 14:11:08.601088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.601053    4291 cpu_manager.go:414] "containerMap: removing container" podUID="46198189-2833-4782-9772-193ca26c986c" containerName="revoke-keys"
Oct 30 14:11:08.601088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.601075    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46198189-2833-4782-9772-193ca26c986c" containerName="revoke-keys"
Oct 30 14:11:08.601306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.601125    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d7d9f56a-ee59-492a-8f08-239985962d91" containerName="revoke-keys"
Oct 30 14:11:08.601306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.601133    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d7d9f56a-ee59-492a-8f08-239985962d91" containerName="revoke-keys"
Oct 30 14:11:08.601306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.601273    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="46198189-2833-4782-9772-193ca26c986c" containerName="revoke-keys"
Oct 30 14:11:08.601306 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.601296    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d7d9f56a-ee59-492a-8f08-239985962d91" containerName="revoke-keys"
Oct 30 14:11:08.605390 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.605354    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.612986 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.612945    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"]
Oct 30 14:11:08.622530 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.622502    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:11:08.622744 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.622587    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:11:08.623177 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.623149    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:08.623270 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.623228    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:11:08.662174 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.662132    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.662382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.662182    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.662382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.662315    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mlzh\" (UniqueName: \"kubernetes.io/projected/cc6eebc5-4df0-497b-853d-83701f149949-kube-api-access-4mlzh\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.763966 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.763921    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.764175 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.763981    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.764175 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.764043    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mlzh\" (UniqueName: \"kubernetes.io/projected/cc6eebc5-4df0-497b-853d-83701f149949-kube-api-access-4mlzh\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.764627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.764603    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.766405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.766381    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-xks-ca\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.783550 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.783491    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mlzh\" (UniqueName: \"kubernetes.io/projected/cc6eebc5-4df0-497b-853d-83701f149949-kube-api-access-4mlzh\") pod \"maas-api-revoke-sub-308a008f-gf77p\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:08.924029 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:08.923993    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:09.140751 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:09.140715    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:11:09.229832 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:09.229800    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"]
Oct 30 14:11:09.230686 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:11:09.230620    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6eebc5_4df0_497b_853d_83701f149949.slice/crio-99fce86e6a876f9d67586fed20dec81e4d5d8581478a97ecdeaae4a4e2c58840 WatchSource:0}: Error finding container 99fce86e6a876f9d67586fed20dec81e4d5d8581478a97ecdeaae4a4e2c58840: Status 404 returned error can't find the container with id 99fce86e6a876f9d67586fed20dec81e4d5d8581478a97ecdeaae4a4e2c58840
Oct 30 14:11:09.888733 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:09.886229    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ddf1edb-e4c4-45e0-8607-3c19f4076ec0" path="/var/lib/kubelet/pods/8ddf1edb-e4c4-45e0-8607-3c19f4076ec0/volumes"
Oct 30 14:11:10.005656 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:10.005009    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p" event={"ID":"cc6eebc5-4df0-497b-853d-83701f149949","Type":"ContainerStarted","Data":"22f36e3801cea7235fdca731278ab6a1eef6934e5bb25b6e168483f98bac8619"}
Oct 30 14:11:10.005656 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:10.005079    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p" event={"ID":"cc6eebc5-4df0-497b-853d-83701f149949","Type":"ContainerStarted","Data":"99fce86e6a876f9d67586fed20dec81e4d5d8581478a97ecdeaae4a4e2c58840"}
Oct 30 14:11:10.053115 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:10.051708    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p" podStartSLOduration=2.0516886579999998 podStartE2EDuration="2.051688658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:10.047684816 +0000 UTC m=+4080.307923051" watchObservedRunningTime="2026-10-30 14:11:10.051688658 +0000 UTC m=+4080.311926893"
Oct 30 14:11:10.729298 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:10.729268    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"]
Oct 30 14:11:10.740595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:10.740571    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-c3efb2-s6lxz"]
Oct 30 14:11:10.902455 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:10.902410    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" podUID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:11.015629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:11.015546    4291 generic.go:356] "Generic (PLEG): container finished" podID="cc6eebc5-4df0-497b-853d-83701f149949" containerID="22f36e3801cea7235fdca731278ab6a1eef6934e5bb25b6e168483f98bac8619" exitCode=0
Oct 30 14:11:11.015629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:11.015597    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p" event={"ID":"cc6eebc5-4df0-497b-853d-83701f149949","Type":"ContainerDied","Data":"22f36e3801cea7235fdca731278ab6a1eef6934e5bb25b6e168483f98bac8619"}
Oct 30 14:11:11.881770 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:11.881735    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d7d9f56a-ee59-492a-8f08-239985962d91" path="/var/lib/kubelet/pods/d7d9f56a-ee59-492a-8f08-239985962d91/volumes"
Oct 30 14:11:11.899220 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:11.899187    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" podUID="7fc89d43-2b87-4184-904e-8d86482b70a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:12.175821 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.175786    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:12.186600 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.186545    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" podUID="da77146e-4068-48f5-a507-addb8019e6f0" containerName="payload-processing" containerID="cri-o://ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759" gracePeriod=25
Oct 30 14:11:12.358983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.358955    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc6eebc5-4df0-497b-853d-83701f149949-kube-api-access-4mlzh\" (UniqueName: \"kubernetes.io/projected/cc6eebc5-4df0-497b-853d-83701f149949-kube-api-access-4mlzh\") pod \"cc6eebc5-4df0-497b-853d-83701f149949\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") "
Oct 30 14:11:12.359173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.359025    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-xks-ca\") pod \"cc6eebc5-4df0-497b-853d-83701f149949\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") "
Oct 30 14:11:12.359173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.359096    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-openshift-ca\") pod \"cc6eebc5-4df0-497b-853d-83701f149949\" (UID: \"cc6eebc5-4df0-497b-853d-83701f149949\") "
Oct 30 14:11:12.359509 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.359484    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-openshift-ca" pod "cc6eebc5-4df0-497b-853d-83701f149949" (UID: "cc6eebc5-4df0-497b-853d-83701f149949"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:12.359660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.359623    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:12.361267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.361241    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-xks-ca" pod "cc6eebc5-4df0-497b-853d-83701f149949" (UID: "cc6eebc5-4df0-497b-853d-83701f149949"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:12.361363 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.361274    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc6eebc5-4df0-497b-853d-83701f149949-kube-api-access-4mlzh" pod "cc6eebc5-4df0-497b-853d-83701f149949" (UID: "cc6eebc5-4df0-497b-853d-83701f149949"). InnerVolumeSpecName "kube-api-access-4mlzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:12.460721 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.460619    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mlzh\" (UniqueName: \"kubernetes.io/projected/cc6eebc5-4df0-497b-853d-83701f149949-kube-api-access-4mlzh\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:12.460721 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:12.460673    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc6eebc5-4df0-497b-853d-83701f149949-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:13.031968 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:13.031937    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p"
Oct 30 14:11:13.032404 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:13.031968    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-308a008f-gf77p" event={"ID":"cc6eebc5-4df0-497b-853d-83701f149949","Type":"ContainerDied","Data":"99fce86e6a876f9d67586fed20dec81e4d5d8581478a97ecdeaae4a4e2c58840"}
Oct 30 14:11:13.032404 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:13.032016    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99fce86e6a876f9d67586fed20dec81e4d5d8581478a97ecdeaae4a4e2c58840"
Oct 30 14:11:14.637161 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:14.637131    4291 scope.go:122] "RemoveContainer" containerID="337af867a438c6a0598ec76c2814def90da059dffcec725c5092240fab533d0c"
Oct 30 14:11:14.647259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:14.647239    4291 scope.go:122] "RemoveContainer" containerID="b26d1a5b9856d76ef26c97433f3bb4c716b97e7e88f0adf9bf78713e86cdc93e"
Oct 30 14:11:14.660423 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:14.660403    4291 scope.go:122] "RemoveContainer" containerID="fa86c13b397e233d1a386b7bfc7e16765e41c652db5cfca039f0594137e4aa4c"
Oct 30 14:11:15.908088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:15.908047    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"]
Oct 30 14:11:15.908842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:15.908808    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy" containerID="cri-o://f29254231edf6c54cd248b27171aaab8bfa765c24f65aa89ea8d910608ad4a2c" gracePeriod=30
Oct 30 14:11:16.465474 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.465441    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"]
Oct 30 14:11:16.466232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.466207    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cc6eebc5-4df0-497b-853d-83701f149949" containerName="revoke-keys"
Oct 30 14:11:16.466232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.466230    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc6eebc5-4df0-497b-853d-83701f149949" containerName="revoke-keys"
Oct 30 14:11:16.466464 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.466346    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc6eebc5-4df0-497b-853d-83701f149949" containerName="revoke-keys"
Oct 30 14:11:16.471796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.471778    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.486873 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.486847    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"]
Oct 30 14:11:16.494874 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.494843    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:11:16.495005 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.494941    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:11:16.504431 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.504409    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:16.504546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.504532    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:11:16.520194 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.520165    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.520396 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.520375    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.520581 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.520560    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rst5\" (UniqueName: \"kubernetes.io/projected/04814eef-98e6-4d78-8a41-3d3c0ac05df3-kube-api-access-4rst5\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.521442 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.521414    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" podUID="da77146e-4068-48f5-a507-addb8019e6f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:16.621813 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.621778    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rst5\" (UniqueName: \"kubernetes.io/projected/04814eef-98e6-4d78-8a41-3d3c0ac05df3-kube-api-access-4rst5\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.622036 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.621838    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.622036 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.621966    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.622576 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.622548    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.624369 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.624347    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.641234 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.641212    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rst5\" (UniqueName: \"kubernetes.io/projected/04814eef-98e6-4d78-8a41-3d3c0ac05df3-kube-api-access-4rst5\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:16.786042 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:16.785947    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:17.092964 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:17.092936    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"]
Oct 30 14:11:17.093827 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:11:17.093795    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04814eef_98e6_4d78_8a41_3d3c0ac05df3.slice/crio-5f18be366f2c24813db58f0ca212ec80a445b6460bfb493d976c697f64d6ccd4 WatchSource:0}: Error finding container 5f18be366f2c24813db58f0ca212ec80a445b6460bfb493d976c697f64d6ccd4: Status 404 returned error can't find the container with id 5f18be366f2c24813db58f0ca212ec80a445b6460bfb493d976c697f64d6ccd4
Oct 30 14:11:17.521362 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:17.521324    4291 patch_prober.go:29] interesting pod/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 14:11:17.521532 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:17.521383    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 14:11:18.082727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:18.082695    4291 generic.go:356] "Generic (PLEG): container finished" podID="04814eef-98e6-4d78-8a41-3d3c0ac05df3" containerID="e322a25edd0bbb1993ec057b4132f1537856ab7a9b7a8b442bf5fda434625233" exitCode=0
Oct 30 14:11:18.082901 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:18.082786    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4" event={"ID":"04814eef-98e6-4d78-8a41-3d3c0ac05df3","Type":"ContainerDied","Data":"e322a25edd0bbb1993ec057b4132f1537856ab7a9b7a8b442bf5fda434625233"}
Oct 30 14:11:18.082901 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:18.082839    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4" event={"ID":"04814eef-98e6-4d78-8a41-3d3c0ac05df3","Type":"ContainerStarted","Data":"5f18be366f2c24813db58f0ca212ec80a445b6460bfb493d976c697f64d6ccd4"}
Oct 30 14:11:18.411180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:18.411138    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"]
Oct 30 14:11:18.411707 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:18.411407    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy" containerID="cri-o://8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a" gracePeriod=30
Oct 30 14:11:19.045206 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.045174    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"]
Oct 30 14:11:19.049038 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.049018    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.067594 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.067568    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"]
Oct 30 14:11:19.156263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.156225    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.156441 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.156353    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.156441 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.156385    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/d844cd84-f0d6-4523-989f-ef9c981ed1c7-kube-api-access-9sbm6\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.256595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.256570    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:19.257157 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.257135    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.257254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.257208    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.257254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.257226    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/d844cd84-f0d6-4523-989f-ef9c981ed1c7-kube-api-access-9sbm6\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.257962 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.257937    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.259413 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.259391    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.275795 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.275770    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/d844cd84-f0d6-4523-989f-ef9c981ed1c7-kube-api-access-9sbm6\") pod \"maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.358099 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.358069    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04814eef-98e6-4d78-8a41-3d3c0ac05df3-kube-api-access-4rst5\" (UniqueName: \"kubernetes.io/projected/04814eef-98e6-4d78-8a41-3d3c0ac05df3-kube-api-access-4rst5\") pod \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") "
Oct 30 14:11:19.358286 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.358124    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-xks-ca\") pod \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") "
Oct 30 14:11:19.358286 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.358232    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-openshift-ca\") pod \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\" (UID: \"04814eef-98e6-4d78-8a41-3d3c0ac05df3\") "
Oct 30 14:11:19.358665 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.358612    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-openshift-ca" pod "04814eef-98e6-4d78-8a41-3d3c0ac05df3" (UID: "04814eef-98e6-4d78-8a41-3d3c0ac05df3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:19.360211 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.360185    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-xks-ca" pod "04814eef-98e6-4d78-8a41-3d3c0ac05df3" (UID: "04814eef-98e6-4d78-8a41-3d3c0ac05df3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:19.360296 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.360240    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04814eef-98e6-4d78-8a41-3d3c0ac05df3-kube-api-access-4rst5" pod "04814eef-98e6-4d78-8a41-3d3c0ac05df3" (UID: "04814eef-98e6-4d78-8a41-3d3c0ac05df3"). InnerVolumeSpecName "kube-api-access-4rst5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:19.362375 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.362357    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:19.459991 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.459956    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:19.460350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.459994    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rst5\" (UniqueName: \"kubernetes.io/projected/04814eef-98e6-4d78-8a41-3d3c0ac05df3-kube-api-access-4rst5\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:19.460350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.460012    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04814eef-98e6-4d78-8a41-3d3c0ac05df3-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:19.684416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:19.684335    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"]
Oct 30 14:11:19.689843 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:11:19.689810    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd844cd84_f0d6_4523_989f_ef9c981ed1c7.slice/crio-eb369febf523713aae99554c54a665622076a45ca23f7942c6fb5c6b4e745362 WatchSource:0}: Error finding container eb369febf523713aae99554c54a665622076a45ca23f7942c6fb5c6b4e745362: Status 404 returned error can't find the container with id eb369febf523713aae99554c54a665622076a45ca23f7942c6fb5c6b4e745362
Oct 30 14:11:20.109900 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:20.109873    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"
Oct 30 14:11:20.110092 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:20.109871    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4" event={"ID":"04814eef-98e6-4d78-8a41-3d3c0ac05df3","Type":"ContainerDied","Data":"5f18be366f2c24813db58f0ca212ec80a445b6460bfb493d976c697f64d6ccd4"}
Oct 30 14:11:20.110092 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:20.110012    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5f18be366f2c24813db58f0ca212ec80a445b6460bfb493d976c697f64d6ccd4"
Oct 30 14:11:20.111734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:20.111708    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms" event={"ID":"d844cd84-f0d6-4523-989f-ef9c981ed1c7","Type":"ContainerStarted","Data":"e9ca8ac6723af050ca1c67a1f92e6b7361b79ae5479273f12a49e438e17db869"}
Oct 30 14:11:20.111890 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:20.111758    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms" event={"ID":"d844cd84-f0d6-4523-989f-ef9c981ed1c7","Type":"ContainerStarted","Data":"eb369febf523713aae99554c54a665622076a45ca23f7942c6fb5c6b4e745362"}
Oct 30 14:11:20.150812 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:20.150757    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms" podStartSLOduration=1.150742134 podStartE2EDuration="1.150742134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:20.148374803 +0000 UTC m=+4090.408613041" watchObservedRunningTime="2026-10-30 14:11:20.150742134 +0000 UTC m=+4090.410980382"
Oct 30 14:11:20.890911 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:20.890865    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" podUID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:21.123510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.123475    4291 generic.go:356] "Generic (PLEG): container finished" podID="7beeec12-22df-45bf-8423-6614d0f55596" containerID="f29254231edf6c54cd248b27171aaab8bfa765c24f65aa89ea8d910608ad4a2c" exitCode=0
Oct 30 14:11:21.123698 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.123584    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" event={"ID":"7beeec12-22df-45bf-8423-6614d0f55596","Type":"ContainerDied","Data":"f29254231edf6c54cd248b27171aaab8bfa765c24f65aa89ea8d910608ad4a2c"}
Oct 30 14:11:21.123796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.123695    4291 scope.go:122] "RemoveContainer" containerID="d09969c8c0d6d5d7c37cd7b1768fc6339e7a229b67135d4db9421b8ab9720ce8"
Oct 30 14:11:21.125761 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.125736    4291 generic.go:356] "Generic (PLEG): container finished" podID="d844cd84-f0d6-4523-989f-ef9c981ed1c7" containerID="e9ca8ac6723af050ca1c67a1f92e6b7361b79ae5479273f12a49e438e17db869" exitCode=0
Oct 30 14:11:21.125900 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.125815    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms" event={"ID":"d844cd84-f0d6-4523-989f-ef9c981ed1c7","Type":"ContainerDied","Data":"e9ca8ac6723af050ca1c67a1f92e6b7361b79ae5479273f12a49e438e17db869"}
Oct 30 14:11:21.189345 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.189322    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 14:11:21.283560 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283530    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-istio-token\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-istio-token\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283605    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-certs\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283646    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7beeec12-22df-45bf-8423-6614d0f55596-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beeec12-22df-45bf-8423-6614d0f55596-istiod-ca-cert\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283667    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-socket\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283717    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7beeec12-22df-45bf-8423-6614d0f55596-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beeec12-22df-45bf-8423-6614d0f55596-istio-podinfo\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283764    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-kube-api-access-tk8fz\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-kube-api-access-tk8fz\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283797    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-credential-socket\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283827    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-data\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.283990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283857    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-envoy\") pod \"7beeec12-22df-45bf-8423-6614d0f55596\" (UID: \"7beeec12-22df-45bf-8423-6614d0f55596\") "
Oct 30 14:11:21.284267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.283966    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-certs" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:21.284267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284034    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-socket" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:21.284267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284072    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7beeec12-22df-45bf-8423-6614d0f55596-istiod-ca-cert" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:21.284267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284249    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-data" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:21.284488 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284376    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-credential-socket" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:21.284717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284696    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.284841 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284720    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7beeec12-22df-45bf-8423-6614d0f55596-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.284841 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284734    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.284841 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284747    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.284841 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.284761    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.286052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.286029    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-envoy" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:21.286258 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.286234    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-istio-token" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:21.286353 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.286317    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-kube-api-access-tk8fz" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "kube-api-access-tk8fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:21.286439 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.286420    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7beeec12-22df-45bf-8423-6614d0f55596-istio-podinfo" pod "7beeec12-22df-45bf-8423-6614d0f55596" (UID: "7beeec12-22df-45bf-8423-6614d0f55596"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:11:21.386178 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.386144    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7beeec12-22df-45bf-8423-6614d0f55596-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.386178 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.386171    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.386178 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.386182    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7beeec12-22df-45bf-8423-6614d0f55596-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.386404 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.386192    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tk8fz\" (UniqueName: \"kubernetes.io/projected/7beeec12-22df-45bf-8423-6614d0f55596-kube-api-access-tk8fz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:21.898826 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:21.898795    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" podUID="7fc89d43-2b87-4184-904e-8d86482b70a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:22.133647 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.133607    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"
Oct 30 14:11:22.133823 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.133656    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf" event={"ID":"7beeec12-22df-45bf-8423-6614d0f55596","Type":"ContainerDied","Data":"3673f2ab6abef8f46402381a3ae80c1867ec379f31de9ee6ec3561953d952a5c"}
Oct 30 14:11:22.133823 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.133722    4291 scope.go:122] "RemoveContainer" containerID="f29254231edf6c54cd248b27171aaab8bfa765c24f65aa89ea8d910608ad4a2c"
Oct 30 14:11:22.199839 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.198912    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"]
Oct 30 14:11:22.207833 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.207797    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-98f1cc-openshift-default-6656499bd9-nnwhf"]
Oct 30 14:11:22.292786 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.292761    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:22.396727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.396650    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-openshift-ca\") pod \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") "
Oct 30 14:11:22.396727 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.396708    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d844cd84-f0d6-4523-989f-ef9c981ed1c7-kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/d844cd84-f0d6-4523-989f-ef9c981ed1c7-kube-api-access-9sbm6\") pod \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") "
Oct 30 14:11:22.396930 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.396762    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-xks-ca\") pod \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\" (UID: \"d844cd84-f0d6-4523-989f-ef9c981ed1c7\") "
Oct 30 14:11:22.396974 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.396920    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-openshift-ca" pod "d844cd84-f0d6-4523-989f-ef9c981ed1c7" (UID: "d844cd84-f0d6-4523-989f-ef9c981ed1c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:22.397159 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.397139    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:22.398763 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.398741    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-xks-ca" pod "d844cd84-f0d6-4523-989f-ef9c981ed1c7" (UID: "d844cd84-f0d6-4523-989f-ef9c981ed1c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:22.398845 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.398805    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d844cd84-f0d6-4523-989f-ef9c981ed1c7-kube-api-access-9sbm6" pod "d844cd84-f0d6-4523-989f-ef9c981ed1c7" (UID: "d844cd84-f0d6-4523-989f-ef9c981ed1c7"). InnerVolumeSpecName "kube-api-access-9sbm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:22.498050 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.498022    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sbm6\" (UniqueName: \"kubernetes.io/projected/d844cd84-f0d6-4523-989f-ef9c981ed1c7-kube-api-access-9sbm6\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:22.498050 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:22.498048    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d844cd84-f0d6-4523-989f-ef9c981ed1c7-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.142314 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.142288    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"
Oct 30 14:11:23.142738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.142280    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms" event={"ID":"d844cd84-f0d6-4523-989f-ef9c981ed1c7","Type":"ContainerDied","Data":"eb369febf523713aae99554c54a665622076a45ca23f7942c6fb5c6b4e745362"}
Oct 30 14:11:23.142738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.142404    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb369febf523713aae99554c54a665622076a45ca23f7942c6fb5c6b4e745362"
Oct 30 14:11:23.689518 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.689461    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 14:11:23.824773 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.824738    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-credential-socket\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.824952 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.824785    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-envoy\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.824952 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.824810    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-certs\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.824952 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.824841    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-podinfo\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.824952 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.824925    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-socket\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.825191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.824970    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istiod-ca-cert\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.825191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825006    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-token\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-token\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.825191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825018    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-credential-socket" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:23.825191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825038    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-kube-api-access-skvzk\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-kube-api-access-skvzk\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.825191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825078    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-data\") pod \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\" (UID: \"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a\") "
Oct 30 14:11:23.825191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825163    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-certs" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:23.825535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825268    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-socket" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:23.825535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825461    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istiod-ca-cert" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:23.825691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825557    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-data" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:23.825778 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825752    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.825842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825784    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.825842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825800    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.825842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825814    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.825842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.825827    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.827240 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.827218    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-podinfo" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:11:23.827350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.827285    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-kube-api-access-skvzk" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "kube-api-access-skvzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:23.827452 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.827435    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-token" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:23.827512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.827438    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-envoy" pod "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" (UID: "fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:23.878138 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.878103    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7beeec12-22df-45bf-8423-6614d0f55596" path="/var/lib/kubelet/pods/7beeec12-22df-45bf-8423-6614d0f55596/volumes"
Oct 30 14:11:23.926779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.926742    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.926779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.926770    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skvzk\" (UniqueName: \"kubernetes.io/projected/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-kube-api-access-skvzk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.926779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.926781    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:23.926779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:23.926790    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:24.150710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.150673    4291 generic.go:356] "Generic (PLEG): container finished" podID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerID="8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a" exitCode=0
Oct 30 14:11:24.151128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.150723    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"
Oct 30 14:11:24.151128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.150754    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" event={"ID":"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a","Type":"ContainerDied","Data":"8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a"}
Oct 30 14:11:24.151128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.150808    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6" event={"ID":"fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a","Type":"ContainerDied","Data":"5b79741186f445534753688da032f4d71fb29a73997c8ef132546b5131ee6d4f"}
Oct 30 14:11:24.151128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.150831    4291 scope.go:122] "RemoveContainer" containerID="8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a"
Oct 30 14:11:24.162108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.162090    4291 scope.go:122] "RemoveContainer" containerID="5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38"
Oct 30 14:11:24.171690 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.171671    4291 scope.go:122] "RemoveContainer" containerID="8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a"
Oct 30 14:11:24.171945 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:24.171929    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a\": container with ID starting with 8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a not found: ID does not exist" containerID="8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a"
Oct 30 14:11:24.172009 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.171954    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a"} err="failed to get container status \"8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a\": rpc error: code = NotFound desc = could not find container \"8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a\": container with ID starting with 8f7d94e8c0b6f03d39fb48f8a48d113f794e4964adc8f2a2880c8f38d1b2be6a not found: ID does not exist"
Oct 30 14:11:24.172009 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.171969    4291 scope.go:122] "RemoveContainer" containerID="5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38"
Oct 30 14:11:24.172199 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:24.172180    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38\": container with ID starting with 5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38 not found: ID does not exist" containerID="5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38"
Oct 30 14:11:24.172263 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.172210    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38"} err="failed to get container status \"5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38\": rpc error: code = NotFound desc = could not find container \"5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38\": container with ID starting with 5dc9c18315cb2fa61f3ff944d17e3d5ceb4165c46ace4a11faaaf6dba99a9c38 not found: ID does not exist"
Oct 30 14:11:24.202476 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.202448    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"]
Oct 30 14:11:24.214117 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:24.214093    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-c3efb2-openshift-default-84bdd76dfc-2rcs6"]
Oct 30 14:11:25.879429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:25.879398    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" path="/var/lib/kubelet/pods/fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a/volumes"
Oct 30 14:11:25.977738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:25.977713    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 14:11:26.025153 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.025124    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 14:11:26.060732 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.060696    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc89d43-2b87-4184-904e-8d86482b70a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc89d43-2b87-4184-904e-8d86482b70a6-plugins-config-volume\") pod \"7fc89d43-2b87-4184-904e-8d86482b70a6\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") "
Oct 30 14:11:26.060883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.060859    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc89d43-2b87-4184-904e-8d86482b70a6-kube-api-access-n99np\" (UniqueName: \"kubernetes.io/projected/7fc89d43-2b87-4184-904e-8d86482b70a6-kube-api-access-n99np\") pod \"7fc89d43-2b87-4184-904e-8d86482b70a6\" (UID: \"7fc89d43-2b87-4184-904e-8d86482b70a6\") "
Oct 30 14:11:26.061078 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.061056    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc89d43-2b87-4184-904e-8d86482b70a6-plugins-config-volume" pod "7fc89d43-2b87-4184-904e-8d86482b70a6" (UID: "7fc89d43-2b87-4184-904e-8d86482b70a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:26.061261 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.061241    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fc89d43-2b87-4184-904e-8d86482b70a6-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:26.063374 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.063345    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc89d43-2b87-4184-904e-8d86482b70a6-kube-api-access-n99np" pod "7fc89d43-2b87-4184-904e-8d86482b70a6" (UID: "7fc89d43-2b87-4184-904e-8d86482b70a6"). InnerVolumeSpecName "kube-api-access-n99np". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:26.161841 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.161810    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-plugins-config-volume\") pod \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") "
Oct 30 14:11:26.162035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.162016    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-kube-api-access-jqxp9\" (UniqueName: \"kubernetes.io/projected/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-kube-api-access-jqxp9\") pod \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\" (UID: \"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc\") "
Oct 30 14:11:26.162262 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.162237    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-plugins-config-volume" pod "bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" (UID: "bcc28f6d-ac8b-4142-a4e3-ab628074f4dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:26.162496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.162474    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:26.162496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.162494    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n99np\" (UniqueName: \"kubernetes.io/projected/7fc89d43-2b87-4184-904e-8d86482b70a6-kube-api-access-n99np\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:26.164213 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.164187    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-kube-api-access-jqxp9" pod "bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" (UID: "bcc28f6d-ac8b-4142-a4e3-ab628074f4dc"). InnerVolumeSpecName "kube-api-access-jqxp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:26.167167 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.167140    4291 generic.go:356] "Generic (PLEG): container finished" podID="7fc89d43-2b87-4184-904e-8d86482b70a6" containerID="8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c" exitCode=0
Oct 30 14:11:26.167273 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.167184    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"
Oct 30 14:11:26.167273 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.167206    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" event={"ID":"7fc89d43-2b87-4184-904e-8d86482b70a6","Type":"ContainerDied","Data":"8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c"}
Oct 30 14:11:26.167405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.167274    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm" event={"ID":"7fc89d43-2b87-4184-904e-8d86482b70a6","Type":"ContainerDied","Data":"5a0e2af0d68881856c1d4ab8d29a3002731b673f15e351d4f01015a2d90a1c85"}
Oct 30 14:11:26.167405 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.167306    4291 scope.go:122] "RemoveContainer" containerID="8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c"
Oct 30 14:11:26.168978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.168951    4291 generic.go:356] "Generic (PLEG): container finished" podID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" containerID="1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d" exitCode=0
Oct 30 14:11:26.169090 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.169002    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"
Oct 30 14:11:26.169090 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.169014    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" event={"ID":"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc","Type":"ContainerDied","Data":"1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d"}
Oct 30 14:11:26.169090 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.169057    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8" event={"ID":"bcc28f6d-ac8b-4142-a4e3-ab628074f4dc","Type":"ContainerDied","Data":"b82da46406ccff7b1aae7bd116b473ff1c12098b8d862ce57064c0e4cf8e241d"}
Oct 30 14:11:26.177562 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.177543    4291 scope.go:122] "RemoveContainer" containerID="8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c"
Oct 30 14:11:26.177863 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:26.177839    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c\": container with ID starting with 8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c not found: ID does not exist" containerID="8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c"
Oct 30 14:11:26.177938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.177871    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c"} err="failed to get container status \"8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c\": rpc error: code = NotFound desc = could not find container \"8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c\": container with ID starting with 8eb3037491e08622ef92a21880c377c4a1d56cdb27f2e4aed22f40bb321e825c not found: ID does not exist"
Oct 30 14:11:26.177938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.177895    4291 scope.go:122] "RemoveContainer" containerID="1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d"
Oct 30 14:11:26.187501 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.187481    4291 scope.go:122] "RemoveContainer" containerID="1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d"
Oct 30 14:11:26.187783 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:26.187761    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d\": container with ID starting with 1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d not found: ID does not exist" containerID="1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d"
Oct 30 14:11:26.187867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.187791    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d"} err="failed to get container status \"1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d\": rpc error: code = NotFound desc = could not find container \"1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d\": container with ID starting with 1478bf683c07c61e9c03e8b8772392e92a8dcd30e91b876a6d5cb52a84a7417d not found: ID does not exist"
Oct 30 14:11:26.248512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.247744    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"]
Oct 30 14:11:26.256199 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.254234    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-98f1cc-688df649c9-d4zq8"]
Oct 30 14:11:26.266563 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.266519    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqxp9\" (UniqueName: \"kubernetes.io/projected/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc-kube-api-access-jqxp9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:26.276226 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.276194    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"]
Oct 30 14:11:26.279201 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.279179    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-98f1cc-7867cb8db6-c2jbm"]
Oct 30 14:11:26.522685 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:26.522579    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" podUID="da77146e-4068-48f5-a507-addb8019e6f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:11:27.884000 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:27.882457    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc89d43-2b87-4184-904e-8d86482b70a6" path="/var/lib/kubelet/pods/7fc89d43-2b87-4184-904e-8d86482b70a6/volumes"
Oct 30 14:11:27.884000 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:27.883127    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" path="/var/lib/kubelet/pods/bcc28f6d-ac8b-4142-a4e3-ab628074f4dc/volumes"
Oct 30 14:11:30.093277 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:30.093244    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:11:32.010651 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.010607    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"]
Oct 30 14:11:32.011174 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011158    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7fc89d43-2b87-4184-904e-8d86482b70a6" containerName="payload-processing"
Oct 30 14:11:32.011232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011176    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc89d43-2b87-4184-904e-8d86482b70a6" containerName="payload-processing"
Oct 30 14:11:32.011232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011189    4291 cpu_manager.go:414] "containerMap: removing container" podUID="04814eef-98e6-4d78-8a41-3d3c0ac05df3" containerName="revoke-keys"
Oct 30 14:11:32.011232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011194    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04814eef-98e6-4d78-8a41-3d3c0ac05df3" containerName="revoke-keys"
Oct 30 14:11:32.011232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011210    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy"
Oct 30 14:11:32.011232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011215    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy"
Oct 30 14:11:32.011232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011233    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d844cd84-f0d6-4523-989f-ef9c981ed1c7" containerName="revoke-keys"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011238    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d844cd84-f0d6-4523-989f-ef9c981ed1c7" containerName="revoke-keys"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011260    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011265    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011273    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011280    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011289    4291 cpu_manager.go:414] "containerMap: removing container" podUID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" containerName="payload-pre-processing"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011294    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" containerName="payload-pre-processing"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011364    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011373    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011380    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7beeec12-22df-45bf-8423-6614d0f55596" containerName="istio-proxy"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011386    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcc28f6d-ac8b-4142-a4e3-ab628074f4dc" containerName="payload-pre-processing"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011392    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc89d43-2b87-4184-904e-8d86482b70a6" containerName="payload-processing"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011398    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="04814eef-98e6-4d78-8a41-3d3c0ac05df3" containerName="revoke-keys"
Oct 30 14:11:32.011432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.011406    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d844cd84-f0d6-4523-989f-ef9c981ed1c7" containerName="revoke-keys"
Oct 30 14:11:32.016512 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.016486    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.031877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.031842    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"]
Oct 30 14:11:32.036510 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.036469    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:32.038239 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.038211    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:11:32.038446 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.038304    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:11:32.130794 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.130764    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:11:32.143442 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.143416    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae194-a940-4a10-b612-6372a1774e53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.143583 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.143564    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jv6cp\" (UniqueName: \"kubernetes.io/projected/83eae194-a940-4a10-b612-6372a1774e53-kube-api-access-jv6cp\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.143663 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.143603    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae194-a940-4a10-b612-6372a1774e53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.244602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.244572    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae194-a940-4a10-b612-6372a1774e53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.244763 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.244714    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jv6cp\" (UniqueName: \"kubernetes.io/projected/83eae194-a940-4a10-b612-6372a1774e53-kube-api-access-jv6cp\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.244763 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.244747    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae194-a940-4a10-b612-6372a1774e53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.245421 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.245399    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae194-a940-4a10-b612-6372a1774e53-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.246995 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.246973    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae194-a940-4a10-b612-6372a1774e53-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.265849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.265787    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jv6cp\" (UniqueName: \"kubernetes.io/projected/83eae194-a940-4a10-b612-6372a1774e53-kube-api-access-jv6cp\") pod \"maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.334202 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.334172    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:32.462270 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.462238    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 14:11:32.548109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.548018    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da77146e-4068-48f5-a507-addb8019e6f0-kube-api-access-l9k9j\" (UniqueName: \"kubernetes.io/projected/da77146e-4068-48f5-a507-addb8019e6f0-kube-api-access-l9k9j\") pod \"da77146e-4068-48f5-a507-addb8019e6f0\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") "
Oct 30 14:11:32.548305 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.548199    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da77146e-4068-48f5-a507-addb8019e6f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da77146e-4068-48f5-a507-addb8019e6f0-plugins-config-volume\") pod \"da77146e-4068-48f5-a507-addb8019e6f0\" (UID: \"da77146e-4068-48f5-a507-addb8019e6f0\") "
Oct 30 14:11:32.548730 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.548623    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da77146e-4068-48f5-a507-addb8019e6f0-plugins-config-volume" pod "da77146e-4068-48f5-a507-addb8019e6f0" (UID: "da77146e-4068-48f5-a507-addb8019e6f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:32.550723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.550690    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da77146e-4068-48f5-a507-addb8019e6f0-kube-api-access-l9k9j" pod "da77146e-4068-48f5-a507-addb8019e6f0" (UID: "da77146e-4068-48f5-a507-addb8019e6f0"). InnerVolumeSpecName "kube-api-access-l9k9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:32.642073 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.642045    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"]
Oct 30 14:11:32.643945 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:11:32.643918    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eae194_a940_4a10_b612_6372a1774e53.slice/crio-07be0e15b7d550db599ba47ab2b973892b6b4e63db552d307b33fc4141483251 WatchSource:0}: Error finding container 07be0e15b7d550db599ba47ab2b973892b6b4e63db552d307b33fc4141483251: Status 404 returned error can't find the container with id 07be0e15b7d550db599ba47ab2b973892b6b4e63db552d307b33fc4141483251
Oct 30 14:11:32.649976 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.649954    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9k9j\" (UniqueName: \"kubernetes.io/projected/da77146e-4068-48f5-a507-addb8019e6f0-kube-api-access-l9k9j\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:32.650050 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:32.649978    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/da77146e-4068-48f5-a507-addb8019e6f0-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:33.240779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.240749    4291 generic.go:356] "Generic (PLEG): container finished" podID="da77146e-4068-48f5-a507-addb8019e6f0" containerID="ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759" exitCode=0
Oct 30 14:11:33.241280 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.240806    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"
Oct 30 14:11:33.241280 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.240837    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" event={"ID":"da77146e-4068-48f5-a507-addb8019e6f0","Type":"ContainerDied","Data":"ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759"}
Oct 30 14:11:33.241280 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.240893    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t" event={"ID":"da77146e-4068-48f5-a507-addb8019e6f0","Type":"ContainerDied","Data":"99bfdefdc5a6fee28a3e4e1dce7a0717e122517e6fbe8b7c032907af19266d39"}
Oct 30 14:11:33.241280 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.240916    4291 scope.go:122] "RemoveContainer" containerID="ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759"
Oct 30 14:11:33.242881 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.242853    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs" event={"ID":"83eae194-a940-4a10-b612-6372a1774e53","Type":"ContainerStarted","Data":"cb7c58e3777d390fb412c4a50c7499369a840ac9b1f4e87e0726435b465bb3dc"}
Oct 30 14:11:33.243029 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.242902    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs" event={"ID":"83eae194-a940-4a10-b612-6372a1774e53","Type":"ContainerStarted","Data":"07be0e15b7d550db599ba47ab2b973892b6b4e63db552d307b33fc4141483251"}
Oct 30 14:11:33.252097 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.252081    4291 scope.go:122] "RemoveContainer" containerID="ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759"
Oct 30 14:11:33.252376 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:33.252355    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759\": container with ID starting with ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759 not found: ID does not exist" containerID="ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759"
Oct 30 14:11:33.252451 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.252387    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759"} err="failed to get container status \"ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759\": rpc error: code = NotFound desc = could not find container \"ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759\": container with ID starting with ef39ee8a4fe4b19f8c26781013ac666b3813eb5a2f79f46b6100c652ef5b2759 not found: ID does not exist"
Oct 30 14:11:33.292879 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.292853    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"]
Oct 30 14:11:33.304423 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.304398    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-c3efb2-d67b546fd-rmp8t"]
Oct 30 14:11:33.332546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.332493    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs" podStartSLOduration=2.332478053 podStartE2EDuration="2.332478053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:33.331690819 +0000 UTC m=+4103.591929056" watchObservedRunningTime="2026-10-30 14:11:33.332478053 +0000 UTC m=+4103.592716287"
Oct 30 14:11:33.878346 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:33.878315    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da77146e-4068-48f5-a507-addb8019e6f0" path="/var/lib/kubelet/pods/da77146e-4068-48f5-a507-addb8019e6f0/volumes"
Oct 30 14:11:34.250662 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:34.250552    4291 generic.go:356] "Generic (PLEG): container finished" podID="83eae194-a940-4a10-b612-6372a1774e53" containerID="cb7c58e3777d390fb412c4a50c7499369a840ac9b1f4e87e0726435b465bb3dc" exitCode=0
Oct 30 14:11:34.251068 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:34.250647    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs" event={"ID":"83eae194-a940-4a10-b612-6372a1774e53","Type":"ContainerDied","Data":"cb7c58e3777d390fb412c4a50c7499369a840ac9b1f4e87e0726435b465bb3dc"}
Oct 30 14:11:35.409805 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.409784    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:35.482615 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.482586    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83eae194-a940-4a10-b612-6372a1774e53-kube-api-access-jv6cp\" (UniqueName: \"kubernetes.io/projected/83eae194-a940-4a10-b612-6372a1774e53-kube-api-access-jv6cp\") pod \"83eae194-a940-4a10-b612-6372a1774e53\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") "
Oct 30 14:11:35.482827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.482626    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83eae194-a940-4a10-b612-6372a1774e53-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae194-a940-4a10-b612-6372a1774e53-maas-api-xks-ca\") pod \"83eae194-a940-4a10-b612-6372a1774e53\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") "
Oct 30 14:11:35.482827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.482750    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83eae194-a940-4a10-b612-6372a1774e53-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae194-a940-4a10-b612-6372a1774e53-maas-api-openshift-ca\") pod \"83eae194-a940-4a10-b612-6372a1774e53\" (UID: \"83eae194-a940-4a10-b612-6372a1774e53\") "
Oct 30 14:11:35.483119 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.483092    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eae194-a940-4a10-b612-6372a1774e53-maas-api-openshift-ca" pod "83eae194-a940-4a10-b612-6372a1774e53" (UID: "83eae194-a940-4a10-b612-6372a1774e53"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:35.483233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.483219    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83eae194-a940-4a10-b612-6372a1774e53-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.484730 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.484694    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83eae194-a940-4a10-b612-6372a1774e53-kube-api-access-jv6cp" pod "83eae194-a940-4a10-b612-6372a1774e53" (UID: "83eae194-a940-4a10-b612-6372a1774e53"). InnerVolumeSpecName "kube-api-access-jv6cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:35.484853 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.484833    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83eae194-a940-4a10-b612-6372a1774e53-maas-api-xks-ca" pod "83eae194-a940-4a10-b612-6372a1774e53" (UID: "83eae194-a940-4a10-b612-6372a1774e53"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:35.584452 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.584364    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jv6cp\" (UniqueName: \"kubernetes.io/projected/83eae194-a940-4a10-b612-6372a1774e53-kube-api-access-jv6cp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:35.584452 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:35.584399    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83eae194-a940-4a10-b612-6372a1774e53-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:36.266690 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:36.266587    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"
Oct 30 14:11:36.266690 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:36.266623    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs" event={"ID":"83eae194-a940-4a10-b612-6372a1774e53","Type":"ContainerDied","Data":"07be0e15b7d550db599ba47ab2b973892b6b4e63db552d307b33fc4141483251"}
Oct 30 14:11:36.266883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:36.266698    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07be0e15b7d550db599ba47ab2b973892b6b4e63db552d307b33fc4141483251"
Oct 30 14:11:36.954115 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:36.954084    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"]
Oct 30 14:11:36.967515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:36.967476    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-a75d02-bq9ms"]
Oct 30 14:11:37.878209 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:37.878173    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d844cd84-f0d6-4523-989f-ef9c981ed1c7" path="/var/lib/kubelet/pods/d844cd84-f0d6-4523-989f-ef9c981ed1c7/volumes"
Oct 30 14:11:38.210279 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:38.210178    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:11:39.277723 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:39.277684    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"]
Oct 30 14:11:39.289426 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:39.289397    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-3f24fd-t6fx4"]
Oct 30 14:11:39.880250 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:39.880216    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04814eef-98e6-4d78-8a41-3d3c0ac05df3" path="/var/lib/kubelet/pods/04814eef-98e6-4d78-8a41-3d3c0ac05df3/volumes"
Oct 30 14:11:45.769373 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:45.769336    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"]
Oct 30 14:11:45.769920 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:45.769605    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy" containerID="cri-o://892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a" gracePeriod=30
Oct 30 14:11:46.358171 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358136    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"]
Oct 30 14:11:46.358653 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358625    4291 cpu_manager.go:414] "containerMap: removing container" podUID="83eae194-a940-4a10-b612-6372a1774e53" containerName="revoke-keys"
Oct 30 14:11:46.358704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358656    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83eae194-a940-4a10-b612-6372a1774e53" containerName="revoke-keys"
Oct 30 14:11:46.358704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358680    4291 cpu_manager.go:414] "containerMap: removing container" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy"
Oct 30 14:11:46.358704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358686    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy"
Oct 30 14:11:46.358704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358696    4291 cpu_manager.go:414] "containerMap: removing container" podUID="da77146e-4068-48f5-a507-addb8019e6f0" containerName="payload-processing"
Oct 30 14:11:46.358704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358701    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da77146e-4068-48f5-a507-addb8019e6f0" containerName="payload-processing"
Oct 30 14:11:46.358867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358786    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd58119f-3ac9-4182-bbbc-48c1ae9fdb9a" containerName="istio-proxy"
Oct 30 14:11:46.358867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358795    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="83eae194-a940-4a10-b612-6372a1774e53" containerName="revoke-keys"
Oct 30 14:11:46.358867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.358804    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="da77146e-4068-48f5-a507-addb8019e6f0" containerName="payload-processing"
Oct 30 14:11:46.363234 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.363217    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.379116 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.379088    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"]
Oct 30 14:11:46.387061 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.387012    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:11:46.387061 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.387050    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:11:46.388796 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.388774    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:46.389268 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.389247    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:11:46.424377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.424338    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.424574 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.424392    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.424885 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.424858    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glbgp\" (UniqueName: \"kubernetes.io/projected/b0002811-c866-40be-b985-dcbe4ee28d94-kube-api-access-glbgp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.525459 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.525427    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glbgp\" (UniqueName: \"kubernetes.io/projected/b0002811-c866-40be-b985-dcbe4ee28d94-kube-api-access-glbgp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.525689 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.525507    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.525689 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.525535    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.526229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.526205    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.527883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.527858    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.545416 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.545391    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glbgp\" (UniqueName: \"kubernetes.io/projected/b0002811-c866-40be-b985-dcbe4ee28d94-kube-api-access-glbgp\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.678182 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.678101    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:46.983685 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:46.983655    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"]
Oct 30 14:11:46.985113 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:11:46.985079    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0002811_c866_40be_b985_dcbe4ee28d94.slice/crio-94f6aaa164d7614cb7deb40cb1f3f16d10108accdb928844ccd6334d5aab082d WatchSource:0}: Error finding container 94f6aaa164d7614cb7deb40cb1f3f16d10108accdb928844ccd6334d5aab082d: Status 404 returned error can't find the container with id 94f6aaa164d7614cb7deb40cb1f3f16d10108accdb928844ccd6334d5aab082d
Oct 30 14:11:47.150880 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:47.150829    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:11:47.385415 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:47.385370    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw" event={"ID":"b0002811-c866-40be-b985-dcbe4ee28d94","Type":"ContainerStarted","Data":"52c722ebe7c1c212053bee47a500f6688c89ce65a8c4adae1b7a47575e53c2e6"}
Oct 30 14:11:47.385582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:47.385436    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw" event={"ID":"b0002811-c866-40be-b985-dcbe4ee28d94","Type":"ContainerStarted","Data":"94f6aaa164d7614cb7deb40cb1f3f16d10108accdb928844ccd6334d5aab082d"}
Oct 30 14:11:47.431975 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:47.431911    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw" podStartSLOduration=1.431892485 podStartE2EDuration="1.431892485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:47.431765306 +0000 UTC m=+4117.692003540" watchObservedRunningTime="2026-10-30 14:11:47.431892485 +0000 UTC m=+4117.692130723"
Oct 30 14:11:48.315590 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.315506    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"]
Oct 30 14:11:48.318376 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.318349    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-cad2c6-6xnqs"]
Oct 30 14:11:48.393142 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.393103    4291 generic.go:356] "Generic (PLEG): container finished" podID="b0002811-c866-40be-b985-dcbe4ee28d94" containerID="52c722ebe7c1c212053bee47a500f6688c89ce65a8c4adae1b7a47575e53c2e6" exitCode=0
Oct 30 14:11:48.393331 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.393186    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw" event={"ID":"b0002811-c866-40be-b985-dcbe4ee28d94","Type":"ContainerDied","Data":"52c722ebe7c1c212053bee47a500f6688c89ce65a8c4adae1b7a47575e53c2e6"}
Oct 30 14:11:48.442422 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.442384    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"]
Oct 30 14:11:48.446460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.446434    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.462806 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.462777    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"]
Oct 30 14:11:48.465539 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.465514    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.465718 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.465555    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.465789 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.465717    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpqqf\" (UniqueName: \"kubernetes.io/projected/c8b2ff24-2e69-4a60-8291-6be1a8e87644-kube-api-access-rpqqf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.566483 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.566396    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.566483 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.566436    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.566483 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.566478    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpqqf\" (UniqueName: \"kubernetes.io/projected/c8b2ff24-2e69-4a60-8291-6be1a8e87644-kube-api-access-rpqqf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.567224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.567199    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.568817 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.568795    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.584863 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.584840    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpqqf\" (UniqueName: \"kubernetes.io/projected/c8b2ff24-2e69-4a60-8291-6be1a8e87644-kube-api-access-rpqqf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:48.762277 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:48.762247    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:49.067630 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.067604    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"]
Oct 30 14:11:49.069163 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:11:49.069104    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b2ff24_2e69_4a60_8291_6be1a8e87644.slice/crio-ed9455c02f31e02c56073f80a2276e070aca5dc68a57dde34b43aec77aa17fce WatchSource:0}: Error finding container ed9455c02f31e02c56073f80a2276e070aca5dc68a57dde34b43aec77aa17fce: Status 404 returned error can't find the container with id ed9455c02f31e02c56073f80a2276e070aca5dc68a57dde34b43aec77aa17fce
Oct 30 14:11:49.401188 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.401154    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z" event={"ID":"c8b2ff24-2e69-4a60-8291-6be1a8e87644","Type":"ContainerStarted","Data":"39220056cfea199cc04f8cf8dd60c26a258f66c2e05ce3a04f117607c2815f06"}
Oct 30 14:11:49.401580 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.401212    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z" event={"ID":"c8b2ff24-2e69-4a60-8291-6be1a8e87644","Type":"ContainerStarted","Data":"ed9455c02f31e02c56073f80a2276e070aca5dc68a57dde34b43aec77aa17fce"}
Oct 30 14:11:49.439319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.439259    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z" podStartSLOduration=1.439246481 podStartE2EDuration="1.439246481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:11:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:11:49.438281713 +0000 UTC m=+4119.698519947" watchObservedRunningTime="2026-10-30 14:11:49.439246481 +0000 UTC m=+4119.699484715"
Oct 30 14:11:49.518674 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.518646    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:49.577358 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.577333    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-xks-ca\") pod \"b0002811-c866-40be-b985-dcbe4ee28d94\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") "
Oct 30 14:11:49.577454 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.577438    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-openshift-ca\") pod \"b0002811-c866-40be-b985-dcbe4ee28d94\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") "
Oct 30 14:11:49.577706 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.577686    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0002811-c866-40be-b985-dcbe4ee28d94-kube-api-access-glbgp\" (UniqueName: \"kubernetes.io/projected/b0002811-c866-40be-b985-dcbe4ee28d94-kube-api-access-glbgp\") pod \"b0002811-c866-40be-b985-dcbe4ee28d94\" (UID: \"b0002811-c866-40be-b985-dcbe4ee28d94\") "
Oct 30 14:11:49.579216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.577893    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-openshift-ca" pod "b0002811-c866-40be-b985-dcbe4ee28d94" (UID: "b0002811-c866-40be-b985-dcbe4ee28d94"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:49.579216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.578098    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:49.580120 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.580097    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-xks-ca" pod "b0002811-c866-40be-b985-dcbe4ee28d94" (UID: "b0002811-c866-40be-b985-dcbe4ee28d94"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:49.580487 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.580466    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0002811-c866-40be-b985-dcbe4ee28d94-kube-api-access-glbgp" pod "b0002811-c866-40be-b985-dcbe4ee28d94" (UID: "b0002811-c866-40be-b985-dcbe4ee28d94"). InnerVolumeSpecName "kube-api-access-glbgp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:49.679617 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.679535    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glbgp\" (UniqueName: \"kubernetes.io/projected/b0002811-c866-40be-b985-dcbe4ee28d94-kube-api-access-glbgp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:49.679617 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.679568    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b0002811-c866-40be-b985-dcbe4ee28d94-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:49.880094 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:49.880062    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83eae194-a940-4a10-b612-6372a1774e53" path="/var/lib/kubelet/pods/83eae194-a940-4a10-b612-6372a1774e53/volumes"
Oct 30 14:11:50.409030 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:50.408994    4291 generic.go:356] "Generic (PLEG): container finished" podID="c8b2ff24-2e69-4a60-8291-6be1a8e87644" containerID="39220056cfea199cc04f8cf8dd60c26a258f66c2e05ce3a04f117607c2815f06" exitCode=0
Oct 30 14:11:50.409431 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:50.409076    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z" event={"ID":"c8b2ff24-2e69-4a60-8291-6be1a8e87644","Type":"ContainerDied","Data":"39220056cfea199cc04f8cf8dd60c26a258f66c2e05ce3a04f117607c2815f06"}
Oct 30 14:11:50.410645 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:50.410610    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"
Oct 30 14:11:50.410780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:50.410614    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw" event={"ID":"b0002811-c866-40be-b985-dcbe4ee28d94","Type":"ContainerDied","Data":"94f6aaa164d7614cb7deb40cb1f3f16d10108accdb928844ccd6334d5aab082d"}
Oct 30 14:11:50.410780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:50.410694    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="94f6aaa164d7614cb7deb40cb1f3f16d10108accdb928844ccd6334d5aab082d"
Oct 30 14:11:51.025496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.025466    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 14:11:51.095459 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.095428    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-socket\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.095691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.095502    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-kube-api-access-vjrs7\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-kube-api-access-vjrs7\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.095691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.095561    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istiod-ca-cert\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.095691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.095586    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-token\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-token\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.095691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.095619    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-podinfo\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.095977 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.095696    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-envoy\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.095977 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.095775    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-socket" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:51.096098 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.096010    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istiod-ca-cert" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:51.096185 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.096149    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-certs\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.096495 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.096426    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-certs" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:51.096717 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.096693    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-data\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.097127 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.096744    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-credential-socket\") pod \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\" (UID: \"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b\") "
Oct 30 14:11:51.097127 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.096979    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-data" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:51.097127 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.097046    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-credential-socket" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:51.097559 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.097539    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.097655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.097565    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.097655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.097579    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.097655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.097593    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.097655 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.097607    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.098433 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.098360    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-token" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:51.098433 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.098400    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-kube-api-access-vjrs7" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "kube-api-access-vjrs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:51.098433 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.098420    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-envoy" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:11:51.098659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.098468    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-podinfo" pod "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" (UID: "ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:11:51.198402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.198366    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.198402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.198399    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjrs7\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-kube-api-access-vjrs7\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.198402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.198412    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.198700 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.198422    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.419956 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.419868    4291 generic.go:356] "Generic (PLEG): container finished" podID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerID="892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a" exitCode=0
Oct 30 14:11:51.419956 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.419917    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"
Oct 30 14:11:51.420420 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.419961    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" event={"ID":"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b","Type":"ContainerDied","Data":"892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a"}
Oct 30 14:11:51.420420 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.420019    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh" event={"ID":"ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b","Type":"ContainerDied","Data":"39076feeeae2482039d9183e6028faeeb6f25c27d909115f6a7149452e3cf885"}
Oct 30 14:11:51.420420 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.420048    4291 scope.go:122] "RemoveContainer" containerID="892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a"
Oct 30 14:11:51.433455 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.433433    4291 scope.go:122] "RemoveContainer" containerID="a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6"
Oct 30 14:11:51.443857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.443833    4291 scope.go:122] "RemoveContainer" containerID="892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a"
Oct 30 14:11:51.444204 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:51.444184    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a\": container with ID starting with 892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a not found: ID does not exist" containerID="892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a"
Oct 30 14:11:51.444283 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.444215    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a"} err="failed to get container status \"892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a\": rpc error: code = NotFound desc = could not find container \"892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a\": container with ID starting with 892e705630fd1b84ed6e47c62d0ec14234d1b44d97c1146e36c675ce1cf0e94a not found: ID does not exist"
Oct 30 14:11:51.444283 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.444234    4291 scope.go:122] "RemoveContainer" containerID="a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6"
Oct 30 14:11:51.444501 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:11:51.444482    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6\": container with ID starting with a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6 not found: ID does not exist" containerID="a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6"
Oct 30 14:11:51.444546 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.444509    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6"} err="failed to get container status \"a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6\": rpc error: code = NotFound desc = could not find container \"a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6\": container with ID starting with a6c17f191fd0313fe9a53da85ce1fa5cb3342fe5708133cfbde7bfc8f758fef6 not found: ID does not exist"
Oct 30 14:11:51.471076 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.471043    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"]
Oct 30 14:11:51.483122 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.483095    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-a75d02-openshift-default-58d7d65745-jnrwh"]
Oct 30 14:11:51.556132 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.556108    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:51.602333 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.602295    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-openshift-ca\") pod \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") "
Oct 30 14:11:51.602495 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.602454    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8b2ff24-2e69-4a60-8291-6be1a8e87644-kube-api-access-rpqqf\" (UniqueName: \"kubernetes.io/projected/c8b2ff24-2e69-4a60-8291-6be1a8e87644-kube-api-access-rpqqf\") pod \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") "
Oct 30 14:11:51.602563 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.602547    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-xks-ca\") pod \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\" (UID: \"c8b2ff24-2e69-4a60-8291-6be1a8e87644\") "
Oct 30 14:11:51.602647 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.602613    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-openshift-ca" pod "c8b2ff24-2e69-4a60-8291-6be1a8e87644" (UID: "c8b2ff24-2e69-4a60-8291-6be1a8e87644"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:51.602991 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.602965    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.604692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.604661    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8b2ff24-2e69-4a60-8291-6be1a8e87644-kube-api-access-rpqqf" pod "c8b2ff24-2e69-4a60-8291-6be1a8e87644" (UID: "c8b2ff24-2e69-4a60-8291-6be1a8e87644"). InnerVolumeSpecName "kube-api-access-rpqqf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:51.604852 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.604832    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-xks-ca" pod "c8b2ff24-2e69-4a60-8291-6be1a8e87644" (UID: "c8b2ff24-2e69-4a60-8291-6be1a8e87644"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:51.703971 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.703892    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpqqf\" (UniqueName: \"kubernetes.io/projected/c8b2ff24-2e69-4a60-8291-6be1a8e87644-kube-api-access-rpqqf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.703971 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.703920    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8b2ff24-2e69-4a60-8291-6be1a8e87644-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:51.878909 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:51.878877    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" path="/var/lib/kubelet/pods/ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b/volumes"
Oct 30 14:11:52.427774 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:52.427737    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"
Oct 30 14:11:52.428342 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:52.427773    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z" event={"ID":"c8b2ff24-2e69-4a60-8291-6be1a8e87644","Type":"ContainerDied","Data":"ed9455c02f31e02c56073f80a2276e070aca5dc68a57dde34b43aec77aa17fce"}
Oct 30 14:11:52.428342 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:52.427859    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed9455c02f31e02c56073f80a2276e070aca5dc68a57dde34b43aec77aa17fce"
Oct 30 14:11:56.573542 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.573497    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"]
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.573983    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c8b2ff24-2e69-4a60-8291-6be1a8e87644" containerName="revoke-keys"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.573994    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8b2ff24-2e69-4a60-8291-6be1a8e87644" containerName="revoke-keys"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574017    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574022    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574030    4291 cpu_manager.go:414] "containerMap: removing container" podUID="b0002811-c866-40be-b985-dcbe4ee28d94" containerName="revoke-keys"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574035    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0002811-c866-40be-b985-dcbe4ee28d94" containerName="revoke-keys"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574047    4291 cpu_manager.go:414] "containerMap: removing container" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574052    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy"
Oct 30 14:11:56.574123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574123    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy"
Oct 30 14:11:56.574683 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574134    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8b2ff24-2e69-4a60-8291-6be1a8e87644" containerName="revoke-keys"
Oct 30 14:11:56.574683 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.574144    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0002811-c866-40be-b985-dcbe4ee28d94" containerName="revoke-keys"
Oct 30 14:11:56.578053 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.578030    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.596535 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.596503    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"]
Oct 30 14:11:56.604077 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.604048    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:11:56.604226 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.604054    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:11:56.606471 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.606444    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:11:56.606582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.606500    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:11:56.671824 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.671774    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.672021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.671896    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffmxp\" (UniqueName: \"kubernetes.io/projected/4fd9ab02-172d-46cc-9253-e2a46b18e63c-kube-api-access-ffmxp\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.672021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.671986    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.773099 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.773058    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.773319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.773120    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffmxp\" (UniqueName: \"kubernetes.io/projected/4fd9ab02-172d-46cc-9253-e2a46b18e63c-kube-api-access-ffmxp\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.773319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.773188    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.773791 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.773766    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.775505 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.775485    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.792361 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.792325    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffmxp\" (UniqueName: \"kubernetes.io/projected/4fd9ab02-172d-46cc-9253-e2a46b18e63c-kube-api-access-ffmxp\") pod \"maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:56.896619 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:56.896582    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:57.236294 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:57.236260    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"]
Oct 30 14:11:57.237301 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:11:57.237274    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd9ab02_172d_46cc_9253_e2a46b18e63c.slice/crio-1e813a38782876c19a67a49f9e64551d9b1c9a5fb4ea6a9baab58bbcbaba53b0 WatchSource:0}: Error finding container 1e813a38782876c19a67a49f9e64551d9b1c9a5fb4ea6a9baab58bbcbaba53b0: Status 404 returned error can't find the container with id 1e813a38782876c19a67a49f9e64551d9b1c9a5fb4ea6a9baab58bbcbaba53b0
Oct 30 14:11:57.464998 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:57.464958    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx" event={"ID":"4fd9ab02-172d-46cc-9253-e2a46b18e63c","Type":"ContainerStarted","Data":"1e813a38782876c19a67a49f9e64551d9b1c9a5fb4ea6a9baab58bbcbaba53b0"}
Oct 30 14:11:58.474629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:58.474502    4291 generic.go:356] "Generic (PLEG): container finished" podID="4fd9ab02-172d-46cc-9253-e2a46b18e63c" containerID="59cdef06c3ec6b7d197c676d86b36900f552e7aee6eef7d98b93d3e903876333" exitCode=0
Oct 30 14:11:58.474629 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:58.474584    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx" event={"ID":"4fd9ab02-172d-46cc-9253-e2a46b18e63c","Type":"ContainerDied","Data":"59cdef06c3ec6b7d197c676d86b36900f552e7aee6eef7d98b93d3e903876333"}
Oct 30 14:11:58.831606 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:58.831505    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"]
Oct 30 14:11:58.861207 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:58.861160    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"]
Oct 30 14:11:59.623616 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.623589    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:11:59.734163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.734069    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-openshift-ca\") pod \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") "
Oct 30 14:11:59.734163 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.734147    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-xks-ca\") pod \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") "
Oct 30 14:11:59.734403 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.734178    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fd9ab02-172d-46cc-9253-e2a46b18e63c-kube-api-access-ffmxp\" (UniqueName: \"kubernetes.io/projected/4fd9ab02-172d-46cc-9253-e2a46b18e63c-kube-api-access-ffmxp\") pod \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\" (UID: \"4fd9ab02-172d-46cc-9253-e2a46b18e63c\") "
Oct 30 14:11:59.734460 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.734401    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-openshift-ca" pod "4fd9ab02-172d-46cc-9253-e2a46b18e63c" (UID: "4fd9ab02-172d-46cc-9253-e2a46b18e63c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:11:59.734597 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.734577    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:59.736325 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.736302    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-xks-ca" pod "4fd9ab02-172d-46cc-9253-e2a46b18e63c" (UID: "4fd9ab02-172d-46cc-9253-e2a46b18e63c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:11:59.736325 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.736314    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd9ab02-172d-46cc-9253-e2a46b18e63c-kube-api-access-ffmxp" pod "4fd9ab02-172d-46cc-9253-e2a46b18e63c" (UID: "4fd9ab02-172d-46cc-9253-e2a46b18e63c"). InnerVolumeSpecName "kube-api-access-ffmxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:11:59.836047 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.836011    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4fd9ab02-172d-46cc-9253-e2a46b18e63c-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:11:59.836047 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:11:59.836042    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffmxp\" (UniqueName: \"kubernetes.io/projected/4fd9ab02-172d-46cc-9253-e2a46b18e63c-kube-api-access-ffmxp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:00.516648 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:00.515621    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx" event={"ID":"4fd9ab02-172d-46cc-9253-e2a46b18e63c","Type":"ContainerDied","Data":"1e813a38782876c19a67a49f9e64551d9b1c9a5fb4ea6a9baab58bbcbaba53b0"}
Oct 30 14:12:00.516648 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:00.515708    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e813a38782876c19a67a49f9e64551d9b1c9a5fb4ea6a9baab58bbcbaba53b0"
Oct 30 14:12:00.516648 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:00.515786    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"
Oct 30 14:12:00.917000 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:00.916958    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"]
Oct 30 14:12:00.924754 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:00.924716    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-51dc9a-cb52z"]
Oct 30 14:12:01.878549 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:01.878509    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b2ff24-2e69-4a60-8291-6be1a8e87644" path="/var/lib/kubelet/pods/c8b2ff24-2e69-4a60-8291-6be1a8e87644/volumes"
Oct 30 14:12:03.832764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:03.832718    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" podUID="cb91e01a-4783-41b2-9306-8ff29c0787f0" containerName="payload-processing" containerID="cri-o://90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1" gracePeriod=25
Oct 30 14:12:03.862518 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:03.862465    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" podUID="dd484118-cad8-4221-b60c-bb9aa540641d" containerName="payload-pre-processing" containerID="cri-o://1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4" gracePeriod=25
Oct 30 14:12:05.534363 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:05.534323    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:12:06.274913 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:06.274875    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" podUID="cb91e01a-4783-41b2-9306-8ff29c0787f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:06.275162 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:06.275133    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" podUID="dd484118-cad8-4221-b60c-bb9aa540641d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:06.961199 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:06.961158    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"]
Oct 30 14:12:07.001386 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:07.001349    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"]
Oct 30 14:12:08.210499 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:08.210459    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:12:09.462003 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.461966    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"]
Oct 30 14:12:09.462466 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.462452    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4fd9ab02-172d-46cc-9253-e2a46b18e63c" containerName="revoke-keys"
Oct 30 14:12:09.462520 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.462467    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fd9ab02-172d-46cc-9253-e2a46b18e63c" containerName="revoke-keys"
Oct 30 14:12:09.462575 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.462566    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fd9ab02-172d-46cc-9253-e2a46b18e63c" containerName="revoke-keys"
Oct 30 14:12:09.462622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.462580    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed20293c-b4b6-4c37-b3b8-eecbaf8eef2b" containerName="istio-proxy"
Oct 30 14:12:09.467296 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.467278    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.482298 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.482271    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"]
Oct 30 14:12:09.490552 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.490525    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:09.490747 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.490727    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:12:09.490871 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.490853    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:12:09.493417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.493398    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:12:09.559055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.559023    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcxwn\" (UniqueName: \"kubernetes.io/projected/d2a00d38-316c-4bdd-a105-b55ca75a40ae-kube-api-access-dcxwn\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.559233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.559062    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.559233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.559153    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.660651 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.660606    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.660853 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.660779    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcxwn\" (UniqueName: \"kubernetes.io/projected/d2a00d38-316c-4bdd-a105-b55ca75a40ae-kube-api-access-dcxwn\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.660853 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.660809    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.661302 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.661282    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.663162 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.663138    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.680174 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.680143    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcxwn\" (UniqueName: \"kubernetes.io/projected/d2a00d38-316c-4bdd-a105-b55ca75a40ae-kube-api-access-dcxwn\") pod \"maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:09.781051 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:09.780961    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:10.082171 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:10.082140    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"]
Oct 30 14:12:10.083731 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:12:10.083700    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a00d38_316c_4bdd_a105_b55ca75a40ae.slice/crio-4df6d9d56026605daff0c4b020e99aae324850a302dfd09f77766a7f9940f769 WatchSource:0}: Error finding container 4df6d9d56026605daff0c4b020e99aae324850a302dfd09f77766a7f9940f769: Status 404 returned error can't find the container with id 4df6d9d56026605daff0c4b020e99aae324850a302dfd09f77766a7f9940f769
Oct 30 14:12:10.639228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:10.639139    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q" event={"ID":"d2a00d38-316c-4bdd-a105-b55ca75a40ae","Type":"ContainerStarted","Data":"4ce63af7c37f50680db2a546f004a09018151972ca19b23a88a1cf483eea9a5a"}
Oct 30 14:12:10.639228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:10.639200    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q" event={"ID":"d2a00d38-316c-4bdd-a105-b55ca75a40ae","Type":"ContainerStarted","Data":"4df6d9d56026605daff0c4b020e99aae324850a302dfd09f77766a7f9940f769"}
Oct 30 14:12:11.537921 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:11.537852    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q" podStartSLOduration=2.537835961 podStartE2EDuration="2.537835961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:10.688120309 +0000 UTC m=+4140.948358540" watchObservedRunningTime="2026-10-30 14:12:11.537835961 +0000 UTC m=+4141.798074195"
Oct 30 14:12:11.539681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:11.539654    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"]
Oct 30 14:12:11.544317 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:11.544298    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-f7ee8a-w5nlw"]
Oct 30 14:12:11.654400 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:11.654355    4291 generic.go:356] "Generic (PLEG): container finished" podID="d2a00d38-316c-4bdd-a105-b55ca75a40ae" containerID="4ce63af7c37f50680db2a546f004a09018151972ca19b23a88a1cf483eea9a5a" exitCode=0
Oct 30 14:12:11.655162 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:11.655132    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q" event={"ID":"d2a00d38-316c-4bdd-a105-b55ca75a40ae","Type":"ContainerDied","Data":"4ce63af7c37f50680db2a546f004a09018151972ca19b23a88a1cf483eea9a5a"}
Oct 30 14:12:11.884703 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:11.884657    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0002811-c866-40be-b985-dcbe4ee28d94" path="/var/lib/kubelet/pods/b0002811-c866-40be-b985-dcbe4ee28d94/volumes"
Oct 30 14:12:11.962475 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:11.962430    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" podUID="604135f2-4b64-4c25-980c-df8d4315a28e" containerName="payload-processing" containerID="cri-o://c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74" gracePeriod=25
Oct 30 14:12:12.002504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.002456    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" podUID="cbb3f7ed-f161-49d2-8654-a4f45e615352" containerName="payload-pre-processing" containerID="cri-o://b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2" gracePeriod=25
Oct 30 14:12:12.808619 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.808595    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:12.905435 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.905399    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-xks-ca\") pod \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") "
Oct 30 14:12:12.905611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.905523    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2a00d38-316c-4bdd-a105-b55ca75a40ae-kube-api-access-dcxwn\" (UniqueName: \"kubernetes.io/projected/d2a00d38-316c-4bdd-a105-b55ca75a40ae-kube-api-access-dcxwn\") pod \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") "
Oct 30 14:12:12.905611 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.905562    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-openshift-ca\") pod \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\" (UID: \"d2a00d38-316c-4bdd-a105-b55ca75a40ae\") "
Oct 30 14:12:12.905997 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.905968    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-openshift-ca" pod "d2a00d38-316c-4bdd-a105-b55ca75a40ae" (UID: "d2a00d38-316c-4bdd-a105-b55ca75a40ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:12.907548 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.907525    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-xks-ca" pod "d2a00d38-316c-4bdd-a105-b55ca75a40ae" (UID: "d2a00d38-316c-4bdd-a105-b55ca75a40ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:12.907660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:12.907584    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2a00d38-316c-4bdd-a105-b55ca75a40ae-kube-api-access-dcxwn" pod "d2a00d38-316c-4bdd-a105-b55ca75a40ae" (UID: "d2a00d38-316c-4bdd-a105-b55ca75a40ae"). InnerVolumeSpecName "kube-api-access-dcxwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:13.007527 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:13.007488    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:13.007527 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:13.007519    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcxwn\" (UniqueName: \"kubernetes.io/projected/d2a00d38-316c-4bdd-a105-b55ca75a40ae-kube-api-access-dcxwn\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:13.007527 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:13.007529    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2a00d38-316c-4bdd-a105-b55ca75a40ae-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:13.415518 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:13.415475    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" podUID="604135f2-4b64-4c25-980c-df8d4315a28e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:13.669569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:13.669487    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"
Oct 30 14:12:13.669569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:13.669519    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q" event={"ID":"d2a00d38-316c-4bdd-a105-b55ca75a40ae","Type":"ContainerDied","Data":"4df6d9d56026605daff0c4b020e99aae324850a302dfd09f77766a7f9940f769"}
Oct 30 14:12:13.669782 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:13.669571    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4df6d9d56026605daff0c4b020e99aae324850a302dfd09f77766a7f9940f769"
Oct 30 14:12:14.191031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:14.190977    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"]
Oct 30 14:12:14.196373 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:14.196347    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"]
Oct 30 14:12:14.210139 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:14.210084    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"]
Oct 30 14:12:14.422939 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:14.422895    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" podUID="cbb3f7ed-f161-49d2-8654-a4f45e615352" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:14.697473 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:14.697441    4291 scope.go:122] "RemoveContainer" containerID="7d2b636a9e486f622dd0d65c8ffc703e385ab6afd39a9e26500c60b141499f11"
Oct 30 14:12:16.274956 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:16.274914    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" podUID="cb91e01a-4783-41b2-9306-8ff29c0787f0" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:16.275369 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:16.275163    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" podUID="dd484118-cad8-4221-b60c-bb9aa540641d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:18.639063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:18.639010    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:12:20.114852 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.114811    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"]
Oct 30 14:12:20.115344 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.115327    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d2a00d38-316c-4bdd-a105-b55ca75a40ae" containerName="revoke-keys"
Oct 30 14:12:20.115394 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.115345    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2a00d38-316c-4bdd-a105-b55ca75a40ae" containerName="revoke-keys"
Oct 30 14:12:20.115437 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.115429    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2a00d38-316c-4bdd-a105-b55ca75a40ae" containerName="revoke-keys"
Oct 30 14:12:20.125817 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.125791    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.138037 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.138008    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"]
Oct 30 14:12:20.144842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.144818    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:20.145049 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.144840    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:12:20.145162 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.144903    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:12:20.153435 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.153399    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:12:20.199882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.199839    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8x7t\" (UniqueName: \"kubernetes.io/projected/4ccbef97-06be-42b9-9edd-36db0c2922e9-kube-api-access-x8x7t\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.200086 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.200054    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.200233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.200212    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.301960 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.301914    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8x7t\" (UniqueName: \"kubernetes.io/projected/4ccbef97-06be-42b9-9edd-36db0c2922e9-kube-api-access-x8x7t\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.302134 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.302057    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.302179 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.302146    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.302749 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.302727    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.304715 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.304692    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.321319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.321288    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8x7t\" (UniqueName: \"kubernetes.io/projected/4ccbef97-06be-42b9-9edd-36db0c2922e9-kube-api-access-x8x7t\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.444393 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.444309    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:20.813673 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.813576    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"]
Oct 30 14:12:20.822986 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.822956    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-844856-8bsbx"]
Oct 30 14:12:20.864010 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:20.863914    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"]
Oct 30 14:12:21.727826 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:21.727775    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht" event={"ID":"4ccbef97-06be-42b9-9edd-36db0c2922e9","Type":"ContainerStarted","Data":"e66e950494034f3b8afc2a9011139ce74f689cbb84e42ac773b30f5c51590f7a"}
Oct 30 14:12:21.728308 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:21.727850    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht" event={"ID":"4ccbef97-06be-42b9-9edd-36db0c2922e9","Type":"ContainerStarted","Data":"557c68840cd7d4b5f58f0ce1de9c58fd03ab7afaa62370cc088db610d9695ba2"}
Oct 30 14:12:21.768945 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:21.768895    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht" podStartSLOduration=1.768880705 podStartE2EDuration="1.768880705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:21.765493979 +0000 UTC m=+4152.025732215" watchObservedRunningTime="2026-10-30 14:12:21.768880705 +0000 UTC m=+4152.029118939"
Oct 30 14:12:21.879142 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:21.879106    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fd9ab02-172d-46cc-9253-e2a46b18e63c" path="/var/lib/kubelet/pods/4fd9ab02-172d-46cc-9253-e2a46b18e63c/volumes"
Oct 30 14:12:22.756214 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:22.756175    4291 generic.go:356] "Generic (PLEG): container finished" podID="4ccbef97-06be-42b9-9edd-36db0c2922e9" containerID="e66e950494034f3b8afc2a9011139ce74f689cbb84e42ac773b30f5c51590f7a" exitCode=0
Oct 30 14:12:22.756662 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:22.756261    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht" event={"ID":"4ccbef97-06be-42b9-9edd-36db0c2922e9","Type":"ContainerDied","Data":"e66e950494034f3b8afc2a9011139ce74f689cbb84e42ac773b30f5c51590f7a"}
Oct 30 14:12:23.415735 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:23.415696    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" podUID="604135f2-4b64-4c25-980c-df8d4315a28e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:23.995087 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:23.995052    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:24.046204 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.046169    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccbef97-06be-42b9-9edd-36db0c2922e9-kube-api-access-x8x7t\" (UniqueName: \"kubernetes.io/projected/4ccbef97-06be-42b9-9edd-36db0c2922e9-kube-api-access-x8x7t\") pod \"4ccbef97-06be-42b9-9edd-36db0c2922e9\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") "
Oct 30 14:12:24.046378 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.046321    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-xks-ca\") pod \"4ccbef97-06be-42b9-9edd-36db0c2922e9\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") "
Oct 30 14:12:24.046378 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.046360    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-openshift-ca\") pod \"4ccbef97-06be-42b9-9edd-36db0c2922e9\" (UID: \"4ccbef97-06be-42b9-9edd-36db0c2922e9\") "
Oct 30 14:12:24.047187 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.046979    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-openshift-ca" pod "4ccbef97-06be-42b9-9edd-36db0c2922e9" (UID: "4ccbef97-06be-42b9-9edd-36db0c2922e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:24.047187 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.047158    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:24.050055 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.049999    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-xks-ca" pod "4ccbef97-06be-42b9-9edd-36db0c2922e9" (UID: "4ccbef97-06be-42b9-9edd-36db0c2922e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:24.051370 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.051347    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccbef97-06be-42b9-9edd-36db0c2922e9-kube-api-access-x8x7t" pod "4ccbef97-06be-42b9-9edd-36db0c2922e9" (UID: "4ccbef97-06be-42b9-9edd-36db0c2922e9"). InnerVolumeSpecName "kube-api-access-x8x7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:24.143561 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.143536    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 14:12:24.147947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.147920    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8x7t\" (UniqueName: \"kubernetes.io/projected/4ccbef97-06be-42b9-9edd-36db0c2922e9-kube-api-access-x8x7t\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:24.147947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.147946    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4ccbef97-06be-42b9-9edd-36db0c2922e9-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:24.150742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.150721    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 14:12:24.249096 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249057    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd484118-cad8-4221-b60c-bb9aa540641d-kube-api-access-l7j68\" (UniqueName: \"kubernetes.io/projected/dd484118-cad8-4221-b60c-bb9aa540641d-kube-api-access-l7j68\") pod \"dd484118-cad8-4221-b60c-bb9aa540641d\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") "
Oct 30 14:12:24.249096 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249107    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb91e01a-4783-41b2-9306-8ff29c0787f0-kube-api-access-gpvcq\" (UniqueName: \"kubernetes.io/projected/cb91e01a-4783-41b2-9306-8ff29c0787f0-kube-api-access-gpvcq\") pod \"cb91e01a-4783-41b2-9306-8ff29c0787f0\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") "
Oct 30 14:12:24.249400 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249126    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd484118-cad8-4221-b60c-bb9aa540641d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd484118-cad8-4221-b60c-bb9aa540641d-plugins-config-volume\") pod \"dd484118-cad8-4221-b60c-bb9aa540641d\" (UID: \"dd484118-cad8-4221-b60c-bb9aa540641d\") "
Oct 30 14:12:24.249400 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249223    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb91e01a-4783-41b2-9306-8ff29c0787f0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb91e01a-4783-41b2-9306-8ff29c0787f0-plugins-config-volume\") pod \"cb91e01a-4783-41b2-9306-8ff29c0787f0\" (UID: \"cb91e01a-4783-41b2-9306-8ff29c0787f0\") "
Oct 30 14:12:24.249774 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249608    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd484118-cad8-4221-b60c-bb9aa540641d-plugins-config-volume" pod "dd484118-cad8-4221-b60c-bb9aa540641d" (UID: "dd484118-cad8-4221-b60c-bb9aa540641d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:24.249774 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249743    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb91e01a-4783-41b2-9306-8ff29c0787f0-plugins-config-volume" pod "cb91e01a-4783-41b2-9306-8ff29c0787f0" (UID: "cb91e01a-4783-41b2-9306-8ff29c0787f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:24.250012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249881    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cb91e01a-4783-41b2-9306-8ff29c0787f0-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:24.250012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.249901    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dd484118-cad8-4221-b60c-bb9aa540641d-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:24.251424 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.251401    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd484118-cad8-4221-b60c-bb9aa540641d-kube-api-access-l7j68" pod "dd484118-cad8-4221-b60c-bb9aa540641d" (UID: "dd484118-cad8-4221-b60c-bb9aa540641d"). InnerVolumeSpecName "kube-api-access-l7j68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:24.251878 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.251857    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb91e01a-4783-41b2-9306-8ff29c0787f0-kube-api-access-gpvcq" pod "cb91e01a-4783-41b2-9306-8ff29c0787f0" (UID: "cb91e01a-4783-41b2-9306-8ff29c0787f0"). InnerVolumeSpecName "kube-api-access-gpvcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:24.351491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.351459    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7j68\" (UniqueName: \"kubernetes.io/projected/dd484118-cad8-4221-b60c-bb9aa540641d-kube-api-access-l7j68\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:24.351491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.351492    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpvcq\" (UniqueName: \"kubernetes.io/projected/cb91e01a-4783-41b2-9306-8ff29c0787f0-kube-api-access-gpvcq\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:24.423407 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.423366    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" podUID="cbb3f7ed-f161-49d2-8654-a4f45e615352" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:24.772128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.772049    4291 generic.go:356] "Generic (PLEG): container finished" podID="cb91e01a-4783-41b2-9306-8ff29c0787f0" containerID="90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1" exitCode=0
Oct 30 14:12:24.772128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.772092    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"
Oct 30 14:12:24.772319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.772125    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" event={"ID":"cb91e01a-4783-41b2-9306-8ff29c0787f0","Type":"ContainerDied","Data":"90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1"}
Oct 30 14:12:24.772319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.772181    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt" event={"ID":"cb91e01a-4783-41b2-9306-8ff29c0787f0","Type":"ContainerDied","Data":"7cde95daf30631b900a7ca5602b23ebf60a4d35177a73c0b974cb7fe4841cd84"}
Oct 30 14:12:24.772319 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.772207    4291 scope.go:122] "RemoveContainer" containerID="90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1"
Oct 30 14:12:24.773955 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.773936    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"
Oct 30 14:12:24.774071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.773984    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht" event={"ID":"4ccbef97-06be-42b9-9edd-36db0c2922e9","Type":"ContainerDied","Data":"557c68840cd7d4b5f58f0ce1de9c58fd03ab7afaa62370cc088db610d9695ba2"}
Oct 30 14:12:24.774071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.774022    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="557c68840cd7d4b5f58f0ce1de9c58fd03ab7afaa62370cc088db610d9695ba2"
Oct 30 14:12:24.775525 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.775492    4291 generic.go:356] "Generic (PLEG): container finished" podID="dd484118-cad8-4221-b60c-bb9aa540641d" containerID="1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4" exitCode=0
Oct 30 14:12:24.775679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.775528    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"
Oct 30 14:12:24.775679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.775527    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" event={"ID":"dd484118-cad8-4221-b60c-bb9aa540641d","Type":"ContainerDied","Data":"1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4"}
Oct 30 14:12:24.775679 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.775573    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr" event={"ID":"dd484118-cad8-4221-b60c-bb9aa540641d","Type":"ContainerDied","Data":"4a376ec5788fb8b859e0f554e48ce230201b54c5dde9901623a3163de2334f93"}
Oct 30 14:12:24.785260 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.785189    4291 scope.go:122] "RemoveContainer" containerID="90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1"
Oct 30 14:12:24.785564 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:24.785542    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1\": container with ID starting with 90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1 not found: ID does not exist" containerID="90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1"
Oct 30 14:12:24.785701 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.785574    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1"} err="failed to get container status \"90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1\": rpc error: code = NotFound desc = could not find container \"90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1\": container with ID starting with 90688b918a4497cb2fbab030748fc8cac6cea7976157b5259d90ca876366eff1 not found: ID does not exist"
Oct 30 14:12:24.785701 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.785594    4291 scope.go:122] "RemoveContainer" containerID="1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4"
Oct 30 14:12:24.795071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.795051    4291 scope.go:122] "RemoveContainer" containerID="1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4"
Oct 30 14:12:24.795368 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:24.795349    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4\": container with ID starting with 1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4 not found: ID does not exist" containerID="1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4"
Oct 30 14:12:24.795440 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.795379    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4"} err="failed to get container status \"1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4\": rpc error: code = NotFound desc = could not find container \"1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4\": container with ID starting with 1ad21ca2786b8bdb6b8701392068526ec6e49a1ad88e156f7935cfffd247e9e4 not found: ID does not exist"
Oct 30 14:12:24.838942 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.838903    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"]
Oct 30 14:12:24.845569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.845541    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-51dc9a-6cd656bdd-s85qt"]
Oct 30 14:12:24.892172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.892141    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"]
Oct 30 14:12:24.895759 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:24.895731    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-51dc9a-5c44f56dbc-ssgbr"]
Oct 30 14:12:25.877434 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:25.877399    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb91e01a-4783-41b2-9306-8ff29c0787f0" path="/var/lib/kubelet/pods/cb91e01a-4783-41b2-9306-8ff29c0787f0/volumes"
Oct 30 14:12:25.877869 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:25.877811    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd484118-cad8-4221-b60c-bb9aa540641d" path="/var/lib/kubelet/pods/dd484118-cad8-4221-b60c-bb9aa540641d/volumes"
Oct 30 14:12:28.849713 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.849677    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"]
Oct 30 14:12:28.850158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850141    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cb91e01a-4783-41b2-9306-8ff29c0787f0" containerName="payload-processing"
Oct 30 14:12:28.850205 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850159    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb91e01a-4783-41b2-9306-8ff29c0787f0" containerName="payload-processing"
Oct 30 14:12:28.850205 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850171    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4ccbef97-06be-42b9-9edd-36db0c2922e9" containerName="revoke-keys"
Oct 30 14:12:28.850205 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850176    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccbef97-06be-42b9-9edd-36db0c2922e9" containerName="revoke-keys"
Oct 30 14:12:28.850205 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850204    4291 cpu_manager.go:414] "containerMap: removing container" podUID="dd484118-cad8-4221-b60c-bb9aa540641d" containerName="payload-pre-processing"
Oct 30 14:12:28.850347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850210    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd484118-cad8-4221-b60c-bb9aa540641d" containerName="payload-pre-processing"
Oct 30 14:12:28.850347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850288    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb91e01a-4783-41b2-9306-8ff29c0787f0" containerName="payload-processing"
Oct 30 14:12:28.850347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850297    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccbef97-06be-42b9-9edd-36db0c2922e9" containerName="revoke-keys"
Oct 30 14:12:28.850347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.850309    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd484118-cad8-4221-b60c-bb9aa540641d" containerName="payload-pre-processing"
Oct 30 14:12:28.855366 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.855348    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:28.870032 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.870005    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:12:28.872398 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.872377    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"]
Oct 30 14:12:28.879914 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.879893    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:12:28.901782 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.901749    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:28.901918 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.901871    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:28.901991 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.901925    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77fnp\" (UniqueName: \"kubernetes.io/projected/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-kube-api-access-77fnp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:28.936184 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.936146    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:28.936318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:28.936218    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:12:29.002891 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.002856    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:29.003066 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.002940    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:29.003066 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.003004    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77fnp\" (UniqueName: \"kubernetes.io/projected/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-kube-api-access-77fnp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:29.003719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.003698    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:29.005238 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.005218    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:29.021894 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.021861    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77fnp\" (UniqueName: \"kubernetes.io/projected/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-kube-api-access-77fnp\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:29.169026 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.168948    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:29.214447 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.213836    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main" containerID="cri-o://8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3" gracePeriod=45
Oct 30 14:12:29.218168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.217892    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main" containerID="cri-o://53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994" gracePeriod=45
Oct 30 14:12:29.233394 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.233127    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main" containerID="cri-o://1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75" gracePeriod=45
Oct 30 14:12:29.270507 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.270457    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"]
Oct 30 14:12:29.274457 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.274434    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.297443 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.297412    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"]
Oct 30 14:12:29.306301 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.306271    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bd97b\" (UniqueName: \"kubernetes.io/projected/11fa077e-79c2-4ffd-9320-168dd602b772-kube-api-access-bd97b\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.306414 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.306329    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.306471 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.306445    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.409444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.408616    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.409444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.407835    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.409444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.408902    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bd97b\" (UniqueName: \"kubernetes.io/projected/11fa077e-79c2-4ffd-9320-168dd602b772-kube-api-access-bd97b\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.409444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.408948    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.413121 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.412529    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.428768 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.428335    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bd97b\" (UniqueName: \"kubernetes.io/projected/11fa077e-79c2-4ffd-9320-168dd602b772-kube-api-access-bd97b\") pod \"maas-api-revoke-sub-84fda1aa-bmqbc\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.491008 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.490954    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"]
Oct 30 14:12:29.523255 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.523223    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 14:12:29.546044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.546011    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 14:12:29.549960 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.549936    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 14:12:29.610777 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.610745    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tmp-dir\") pod \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") "
Oct 30 14:12:29.610958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.610790    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8c75286-fcfd-4d26-9e75-de16b42a094c-tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c75286-fcfd-4d26-9e75-de16b42a094c-tls-certs\") pod \"c8c75286-fcfd-4d26-9e75-de16b42a094c\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") "
Oct 30 14:12:29.610958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.610819    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-model-cache\") pod \"c8c75286-fcfd-4d26-9e75-de16b42a094c\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") "
Oct 30 14:12:29.610958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.610874    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-dshm\") pod \"4b145b65-81da-48b6-afd6-5ad1ddef2572\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") "
Oct 30 14:12:29.610958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.610899    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-home\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-home\") pod \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") "
Oct 30 14:12:29.610958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.610929    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fe0cce-0dda-410b-bf07-5c5db90c52b8-kube-api-access-fd5zk\" (UniqueName: \"kubernetes.io/projected/78fe0cce-0dda-410b-bf07-5c5db90c52b8-kube-api-access-fd5zk\") pod \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.610965    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-home\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-home\") pod \"c8c75286-fcfd-4d26-9e75-de16b42a094c\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611048    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b145b65-81da-48b6-afd6-5ad1ddef2572-tls-certs\" (UniqueName: \"kubernetes.io/secret/4b145b65-81da-48b6-afd6-5ad1ddef2572-tls-certs\") pod \"4b145b65-81da-48b6-afd6-5ad1ddef2572\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611086    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-dshm\") pod \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611092    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tmp-dir" pod "78fe0cce-0dda-410b-bf07-5c5db90c52b8" (UID: "78fe0cce-0dda-410b-bf07-5c5db90c52b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611121    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-model-cache\") pod \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611155    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-tmp-dir\") pod \"4b145b65-81da-48b6-afd6-5ad1ddef2572\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611206    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-home\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-home\") pod \"4b145b65-81da-48b6-afd6-5ad1ddef2572\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611263    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-model-cache\") pod \"4b145b65-81da-48b6-afd6-5ad1ddef2572\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611305    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-dshm\") pod \"c8c75286-fcfd-4d26-9e75-de16b42a094c\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611337    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8c75286-fcfd-4d26-9e75-de16b42a094c-kube-api-access-pt9nd\" (UniqueName: \"kubernetes.io/projected/c8c75286-fcfd-4d26-9e75-de16b42a094c-kube-api-access-pt9nd\") pod \"c8c75286-fcfd-4d26-9e75-de16b42a094c\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") "
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611341    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-home" pod "c8c75286-fcfd-4d26-9e75-de16b42a094c" (UID: "c8c75286-fcfd-4d26-9e75-de16b42a094c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.611382 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611370    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tls-certs\" (UniqueName: \"kubernetes.io/secret/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tls-certs\") pod \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\" (UID: \"78fe0cce-0dda-410b-bf07-5c5db90c52b8\") "
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611406    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-tmp-dir\") pod \"c8c75286-fcfd-4d26-9e75-de16b42a094c\" (UID: \"c8c75286-fcfd-4d26-9e75-de16b42a094c\") "
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611438    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b145b65-81da-48b6-afd6-5ad1ddef2572-kube-api-access-sd5b4\" (UniqueName: \"kubernetes.io/projected/4b145b65-81da-48b6-afd6-5ad1ddef2572-kube-api-access-sd5b4\") pod \"4b145b65-81da-48b6-afd6-5ad1ddef2572\" (UID: \"4b145b65-81da-48b6-afd6-5ad1ddef2572\") "
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611798    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-model-cache" pod "4b145b65-81da-48b6-afd6-5ad1ddef2572" (UID: "4b145b65-81da-48b6-afd6-5ad1ddef2572"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.611969    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-home" pod "78fe0cce-0dda-410b-bf07-5c5db90c52b8" (UID: "78fe0cce-0dda-410b-bf07-5c5db90c52b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.612028    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.612053    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.612066    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.612254 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.612251    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-model-cache" pod "78fe0cce-0dda-410b-bf07-5c5db90c52b8" (UID: "78fe0cce-0dda-410b-bf07-5c5db90c52b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.612742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.612466    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-tmp-dir" pod "4b145b65-81da-48b6-afd6-5ad1ddef2572" (UID: "4b145b65-81da-48b6-afd6-5ad1ddef2572"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.612742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.612602    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-home" pod "4b145b65-81da-48b6-afd6-5ad1ddef2572" (UID: "4b145b65-81da-48b6-afd6-5ad1ddef2572"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.612866 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.612735    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-model-cache" pod "c8c75286-fcfd-4d26-9e75-de16b42a094c" (UID: "c8c75286-fcfd-4d26-9e75-de16b42a094c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.614870 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.614844    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-dshm" pod "c8c75286-fcfd-4d26-9e75-de16b42a094c" (UID: "c8c75286-fcfd-4d26-9e75-de16b42a094c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.615083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.615059    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-tmp-dir" pod "c8c75286-fcfd-4d26-9e75-de16b42a094c" (UID: "c8c75286-fcfd-4d26-9e75-de16b42a094c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.615191 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.615171    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b145b65-81da-48b6-afd6-5ad1ddef2572-tls-certs" pod "4b145b65-81da-48b6-afd6-5ad1ddef2572" (UID: "4b145b65-81da-48b6-afd6-5ad1ddef2572"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:29.615667 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.615619    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-dshm" pod "78fe0cce-0dda-410b-bf07-5c5db90c52b8" (UID: "78fe0cce-0dda-410b-bf07-5c5db90c52b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.616133 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.616107    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8c75286-fcfd-4d26-9e75-de16b42a094c-tls-certs" pod "c8c75286-fcfd-4d26-9e75-de16b42a094c" (UID: "c8c75286-fcfd-4d26-9e75-de16b42a094c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:29.616867 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.616844    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fe0cce-0dda-410b-bf07-5c5db90c52b8-kube-api-access-fd5zk" pod "78fe0cce-0dda-410b-bf07-5c5db90c52b8" (UID: "78fe0cce-0dda-410b-bf07-5c5db90c52b8"). InnerVolumeSpecName "kube-api-access-fd5zk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:29.617128 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.617101    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8c75286-fcfd-4d26-9e75-de16b42a094c-kube-api-access-pt9nd" pod "c8c75286-fcfd-4d26-9e75-de16b42a094c" (UID: "c8c75286-fcfd-4d26-9e75-de16b42a094c"). InnerVolumeSpecName "kube-api-access-pt9nd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:29.617262 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.617149    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b145b65-81da-48b6-afd6-5ad1ddef2572-kube-api-access-sd5b4" pod "4b145b65-81da-48b6-afd6-5ad1ddef2572" (UID: "4b145b65-81da-48b6-afd6-5ad1ddef2572"). InnerVolumeSpecName "kube-api-access-sd5b4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:29.617335 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.617258    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tls-certs" pod "78fe0cce-0dda-410b-bf07-5c5db90c52b8" (UID: "78fe0cce-0dda-410b-bf07-5c5db90c52b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:29.618023 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.617985    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-dshm" pod "4b145b65-81da-48b6-afd6-5ad1ddef2572" (UID: "4b145b65-81da-48b6-afd6-5ad1ddef2572"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:29.684990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.684890    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:29.713233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713195    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4b145b65-81da-48b6-afd6-5ad1ddef2572-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713228    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713238    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713246    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713254    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713261    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713269    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt9nd\" (UniqueName: \"kubernetes.io/projected/c8c75286-fcfd-4d26-9e75-de16b42a094c-kube-api-access-pt9nd\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713278    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fe0cce-0dda-410b-bf07-5c5db90c52b8-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713287    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713295    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd5b4\" (UniqueName: \"kubernetes.io/projected/4b145b65-81da-48b6-afd6-5ad1ddef2572-kube-api-access-sd5b4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713302    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c8c75286-fcfd-4d26-9e75-de16b42a094c-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713310    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c8c75286-fcfd-4d26-9e75-de16b42a094c-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713319    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4b145b65-81da-48b6-afd6-5ad1ddef2572-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713327    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fe0cce-0dda-410b-bf07-5c5db90c52b8-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.713515 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.713335    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd5zk\" (UniqueName: \"kubernetes.io/projected/78fe0cce-0dda-410b-bf07-5c5db90c52b8-kube-api-access-fd5zk\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:29.815350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.815317    4291 generic.go:356] "Generic (PLEG): container finished" podID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerID="8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3" exitCode=0
Oct 30 14:12:29.815795 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.815755    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" event={"ID":"c8c75286-fcfd-4d26-9e75-de16b42a094c","Type":"ContainerDied","Data":"8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3"}
Oct 30 14:12:29.816071 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.816016    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7" event={"ID":"c8c75286-fcfd-4d26-9e75-de16b42a094c","Type":"ContainerDied","Data":"5959f505adcbb508057a450e2b6469183d8e8733507f5d207d69df6e3f950677"}
Oct 30 14:12:29.816179 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.815774    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"
Oct 30 14:12:29.816536 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.816056    4291 scope.go:122] "RemoveContainer" containerID="8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3"
Oct 30 14:12:29.818758 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.818728    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"
Oct 30 14:12:29.818878 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.818802    4291 generic.go:356] "Generic (PLEG): container finished" podID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerID="53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994" exitCode=0
Oct 30 14:12:29.818878 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.818859    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" event={"ID":"78fe0cce-0dda-410b-bf07-5c5db90c52b8","Type":"ContainerDied","Data":"53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994"}
Oct 30 14:12:29.819014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.818903    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd" event={"ID":"78fe0cce-0dda-410b-bf07-5c5db90c52b8","Type":"ContainerDied","Data":"7b2526d081f2cd61b312b151900ec65be26e5927a252fd56683b231b5e0052fb"}
Oct 30 14:12:29.820800 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.820773    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn" event={"ID":"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3","Type":"ContainerStarted","Data":"4245ddbfe63c3d989fce32201436104360b1dd9ab2b74f1501eed9a85e5e575f"}
Oct 30 14:12:29.820901 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.820822    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn" event={"ID":"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3","Type":"ContainerStarted","Data":"7aec83556ab99a14e8237f64cf0125dbb30dc54d9597dfaa803e5549f4677251"}
Oct 30 14:12:29.822418 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.822390    4291 generic.go:356] "Generic (PLEG): container finished" podID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerID="1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75" exitCode=0
Oct 30 14:12:29.822526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.822437    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"
Oct 30 14:12:29.822526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.822488    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" event={"ID":"4b145b65-81da-48b6-afd6-5ad1ddef2572","Type":"ContainerDied","Data":"1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75"}
Oct 30 14:12:29.822526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.822518    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl" event={"ID":"4b145b65-81da-48b6-afd6-5ad1ddef2572","Type":"ContainerDied","Data":"b3ef40cf69016531d421904fa820e1f883cc31658ff285340c33fbb46fa12a39"}
Oct 30 14:12:29.831693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.831535    4291 scope.go:122] "RemoveContainer" containerID="8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3"
Oct 30 14:12:29.832313 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:29.832281    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3\": container with ID starting with 8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3 not found: ID does not exist" containerID="8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3"
Oct 30 14:12:29.832411 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.832323    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3"} err="failed to get container status \"8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3\": rpc error: code = NotFound desc = could not find container \"8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3\": container with ID starting with 8d319364d4b701f804296182886424c2690b61e5334d9c74bb965ae1ee2e56d3 not found: ID does not exist"
Oct 30 14:12:29.832411 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.832347    4291 scope.go:122] "RemoveContainer" containerID="53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994"
Oct 30 14:12:29.842394 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.842376    4291 scope.go:122] "RemoveContainer" containerID="53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994"
Oct 30 14:12:29.842703 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:29.842684    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994\": container with ID starting with 53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994 not found: ID does not exist" containerID="53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994"
Oct 30 14:12:29.842764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.842710    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994"} err="failed to get container status \"53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994\": rpc error: code = NotFound desc = could not find container \"53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994\": container with ID starting with 53f8877259f3811cad0e7a623ae321c875deac0ab3570fa9d4ce1009a6adc994 not found: ID does not exist"
Oct 30 14:12:29.842764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.842727    4291 scope.go:122] "RemoveContainer" containerID="1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75"
Oct 30 14:12:29.851372 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.851353    4291 scope.go:122] "RemoveContainer" containerID="1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75"
Oct 30 14:12:29.851703 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:29.851608    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75\": container with ID starting with 1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75 not found: ID does not exist" containerID="1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75"
Oct 30 14:12:29.851703 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.851661    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75"} err="failed to get container status \"1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75\": rpc error: code = NotFound desc = could not find container \"1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75\": container with ID starting with 1695414e22af5f598122db2d8c9c635e45b3ddfec651220aad80d7009d6cbf75 not found: ID does not exist"
Oct 30 14:12:29.871622 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.871581    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"]
Oct 30 14:12:29.892532 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.892496    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/distinct-w0-f9ebb5-kserve-b47f7d7f4-d87n7"]
Oct 30 14:12:29.920807 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.920774    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"]
Oct 30 14:12:29.923825 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.923796    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/premium-w0-f9ebb5-kserve-84997cf747-f76wd"]
Oct 30 14:12:29.960599 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.960502    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn" podStartSLOduration=1.960489232 podStartE2EDuration="1.960489232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:29.959861261 +0000 UTC m=+4160.220099492" watchObservedRunningTime="2026-10-30 14:12:29.960489232 +0000 UTC m=+4160.220727466"
Oct 30 14:12:29.990709 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:29.990626    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"]
Oct 30 14:12:29.992061 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:12:29.992033    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11fa077e_79c2_4ffd_9320_168dd602b772.slice/crio-f34b8dd3aa26933b7e2a021f33371649b6c957225c09cf305a5e1a9d3cacb463 WatchSource:0}: Error finding container f34b8dd3aa26933b7e2a021f33371649b6c957225c09cf305a5e1a9d3cacb463: Status 404 returned error can't find the container with id f34b8dd3aa26933b7e2a021f33371649b6c957225c09cf305a5e1a9d3cacb463
Oct 30 14:12:30.014704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.014679    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"]
Oct 30 14:12:30.020496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.020465    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-f9ebb5/simulator-w0-f9ebb5-kserve-68f8c966bb-kshjl"]
Oct 30 14:12:30.525821 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.525729    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"]
Oct 30 14:12:30.551689 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.551651    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"]
Oct 30 14:12:30.833503 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.833415    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc" event={"ID":"11fa077e-79c2-4ffd-9320-168dd602b772","Type":"ContainerStarted","Data":"5ae528bbf15cd638265c4989da9784ab53e29d7f0e8cd1b95cffe22b8ffd58f6"}
Oct 30 14:12:30.833503 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.833485    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc" event={"ID":"11fa077e-79c2-4ffd-9320-168dd602b772","Type":"ContainerStarted","Data":"f34b8dd3aa26933b7e2a021f33371649b6c957225c09cf305a5e1a9d3cacb463"}
Oct 30 14:12:30.835122 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.835094    4291 generic.go:356] "Generic (PLEG): container finished" podID="cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" containerID="4245ddbfe63c3d989fce32201436104360b1dd9ab2b74f1501eed9a85e5e575f" exitCode=0
Oct 30 14:12:30.835237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.835167    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn" event={"ID":"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3","Type":"ContainerDied","Data":"4245ddbfe63c3d989fce32201436104360b1dd9ab2b74f1501eed9a85e5e575f"}
Oct 30 14:12:30.879666 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:30.879601    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc" podStartSLOduration=1.87958575 podStartE2EDuration="1.87958575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:30.876005779 +0000 UTC m=+4161.136244042" watchObservedRunningTime="2026-10-30 14:12:30.87958575 +0000 UTC m=+4161.139823983"
Oct 30 14:12:31.844788 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:31.844754    4291 generic.go:356] "Generic (PLEG): container finished" podID="11fa077e-79c2-4ffd-9320-168dd602b772" containerID="5ae528bbf15cd638265c4989da9784ab53e29d7f0e8cd1b95cffe22b8ffd58f6" exitCode=0
Oct 30 14:12:31.844980 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:31.844841    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc" event={"ID":"11fa077e-79c2-4ffd-9320-168dd602b772","Type":"ContainerDied","Data":"5ae528bbf15cd638265c4989da9784ab53e29d7f0e8cd1b95cffe22b8ffd58f6"}
Oct 30 14:12:31.879136 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:31.879095    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" path="/var/lib/kubelet/pods/4b145b65-81da-48b6-afd6-5ad1ddef2572/volumes"
Oct 30 14:12:31.879811 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:31.879783    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" path="/var/lib/kubelet/pods/78fe0cce-0dda-410b-bf07-5c5db90c52b8/volumes"
Oct 30 14:12:31.880407 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:31.880388    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" path="/var/lib/kubelet/pods/c8c75286-fcfd-4d26-9e75-de16b42a094c/volumes"
Oct 30 14:12:32.033508 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.033474    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"]
Oct 30 14:12:32.036602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.036571    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aed44bf0-5sknv"]
Oct 30 14:12:32.115601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.115572    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:32.242120 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.242091    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-openshift-ca\") pod \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") "
Oct 30 14:12:32.242312 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.242296    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-xks-ca\") pod \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") "
Oct 30 14:12:32.242419 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.242355    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-kube-api-access-77fnp\" (UniqueName: \"kubernetes.io/projected/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-kube-api-access-77fnp\") pod \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\" (UID: \"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3\") "
Oct 30 14:12:32.242560 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.242530    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-openshift-ca" pod "cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" (UID: "cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:32.242746 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.242728    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.249999 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.249968    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-kube-api-access-77fnp" pod "cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" (UID: "cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3"). InnerVolumeSpecName "kube-api-access-77fnp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:32.250377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.250349    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-xks-ca" pod "cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" (UID: "cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:32.320123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.320096    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 14:12:32.322217 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.322196    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 14:12:32.344236 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.344200    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.344236 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.344233    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77fnp\" (UniqueName: \"kubernetes.io/projected/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3-kube-api-access-77fnp\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.444626 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.444541    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbb3f7ed-f161-49d2-8654-a4f45e615352-kube-api-access-px5wx\" (UniqueName: \"kubernetes.io/projected/cbb3f7ed-f161-49d2-8654-a4f45e615352-kube-api-access-px5wx\") pod \"cbb3f7ed-f161-49d2-8654-a4f45e615352\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") "
Oct 30 14:12:32.444805 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.444704    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbb3f7ed-f161-49d2-8654-a4f45e615352-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3f7ed-f161-49d2-8654-a4f45e615352-plugins-config-volume\") pod \"cbb3f7ed-f161-49d2-8654-a4f45e615352\" (UID: \"cbb3f7ed-f161-49d2-8654-a4f45e615352\") "
Oct 30 14:12:32.444805 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.444736    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/604135f2-4b64-4c25-980c-df8d4315a28e-kube-api-access-wxjtb\" (UniqueName: \"kubernetes.io/projected/604135f2-4b64-4c25-980c-df8d4315a28e-kube-api-access-wxjtb\") pod \"604135f2-4b64-4c25-980c-df8d4315a28e\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") "
Oct 30 14:12:32.444805 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.444774    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/604135f2-4b64-4c25-980c-df8d4315a28e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604135f2-4b64-4c25-980c-df8d4315a28e-plugins-config-volume\") pod \"604135f2-4b64-4c25-980c-df8d4315a28e\" (UID: \"604135f2-4b64-4c25-980c-df8d4315a28e\") "
Oct 30 14:12:32.445168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.445141    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbb3f7ed-f161-49d2-8654-a4f45e615352-plugins-config-volume" pod "cbb3f7ed-f161-49d2-8654-a4f45e615352" (UID: "cbb3f7ed-f161-49d2-8654-a4f45e615352"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:32.445247 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.445195    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/604135f2-4b64-4c25-980c-df8d4315a28e-plugins-config-volume" pod "604135f2-4b64-4c25-980c-df8d4315a28e" (UID: "604135f2-4b64-4c25-980c-df8d4315a28e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:32.445292 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.445266    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbb3f7ed-f161-49d2-8654-a4f45e615352-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.446715 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.446686    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbb3f7ed-f161-49d2-8654-a4f45e615352-kube-api-access-px5wx" pod "cbb3f7ed-f161-49d2-8654-a4f45e615352" (UID: "cbb3f7ed-f161-49d2-8654-a4f45e615352"). InnerVolumeSpecName "kube-api-access-px5wx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:32.446830 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.446791    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/604135f2-4b64-4c25-980c-df8d4315a28e-kube-api-access-wxjtb" pod "604135f2-4b64-4c25-980c-df8d4315a28e" (UID: "604135f2-4b64-4c25-980c-df8d4315a28e"). InnerVolumeSpecName "kube-api-access-wxjtb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:32.546385 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.546344    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5wx\" (UniqueName: \"kubernetes.io/projected/cbb3f7ed-f161-49d2-8654-a4f45e615352-kube-api-access-px5wx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.546385 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.546376    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxjtb\" (UniqueName: \"kubernetes.io/projected/604135f2-4b64-4c25-980c-df8d4315a28e-kube-api-access-wxjtb\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.546385 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.546389    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/604135f2-4b64-4c25-980c-df8d4315a28e-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:32.852544 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.852511    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"
Oct 30 14:12:32.852737 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.852550    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn" event={"ID":"cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3","Type":"ContainerDied","Data":"7aec83556ab99a14e8237f64cf0125dbb30dc54d9597dfaa803e5549f4677251"}
Oct 30 14:12:32.852737 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.852610    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7aec83556ab99a14e8237f64cf0125dbb30dc54d9597dfaa803e5549f4677251"
Oct 30 14:12:32.853958 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.853933    4291 generic.go:356] "Generic (PLEG): container finished" podID="cbb3f7ed-f161-49d2-8654-a4f45e615352" containerID="b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2" exitCode=0
Oct 30 14:12:32.854095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.853980    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"
Oct 30 14:12:32.854095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.854011    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" event={"ID":"cbb3f7ed-f161-49d2-8654-a4f45e615352","Type":"ContainerDied","Data":"b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2"}
Oct 30 14:12:32.854095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.854063    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx" event={"ID":"cbb3f7ed-f161-49d2-8654-a4f45e615352","Type":"ContainerDied","Data":"f68c8d79bc4beee1ccd5505fd0102b9a069e3e9327ea7d3c8df51d290e01b6ca"}
Oct 30 14:12:32.854095 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.854078    4291 scope.go:122] "RemoveContainer" containerID="b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2"
Oct 30 14:12:32.855543 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.855520    4291 generic.go:356] "Generic (PLEG): container finished" podID="604135f2-4b64-4c25-980c-df8d4315a28e" containerID="c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74" exitCode=0
Oct 30 14:12:32.855617 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.855556    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"
Oct 30 14:12:32.855719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.855615    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" event={"ID":"604135f2-4b64-4c25-980c-df8d4315a28e","Type":"ContainerDied","Data":"c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74"}
Oct 30 14:12:32.855719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.855684    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt" event={"ID":"604135f2-4b64-4c25-980c-df8d4315a28e","Type":"ContainerDied","Data":"ea8f8fb89f3df8249cff728b75b057ed3b1a75df33a9bcc80d46cf33b2a83872"}
Oct 30 14:12:32.873203 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.873055    4291 scope.go:122] "RemoveContainer" containerID="b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2"
Oct 30 14:12:32.873776 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:32.873737    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2\": container with ID starting with b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2 not found: ID does not exist" containerID="b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2"
Oct 30 14:12:32.873973 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.873787    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2"} err="failed to get container status \"b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2\": rpc error: code = NotFound desc = could not find container \"b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2\": container with ID starting with b22e0f8fe92837340995116fb7c4ab4c55cd810d7b308d1da5a62f6760430ad2 not found: ID does not exist"
Oct 30 14:12:32.873973 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.873826    4291 scope.go:122] "RemoveContainer" containerID="c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74"
Oct 30 14:12:32.887878 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.887840    4291 scope.go:122] "RemoveContainer" containerID="c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74"
Oct 30 14:12:32.888324 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:32.888298    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74\": container with ID starting with c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74 not found: ID does not exist" containerID="c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74"
Oct 30 14:12:32.888399 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.888339    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74"} err="failed to get container status \"c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74\": rpc error: code = NotFound desc = could not find container \"c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74\": container with ID starting with c649b69c1dce59b4b34b7031e5858943984a6e66e41266eee94644a7e0725a74 not found: ID does not exist"
Oct 30 14:12:32.939558 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.939522    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"]
Oct 30 14:12:32.943758 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.943729    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-844856-78b96b74f5-mg7dx"]
Oct 30 14:12:32.980233 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.980200    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"]
Oct 30 14:12:32.987031 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.987009    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-844856-59c9bd7f96-t2bkt"]
Oct 30 14:12:32.996324 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:32.996302    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:33.051784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.051750    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11fa077e-79c2-4ffd-9320-168dd602b772-kube-api-access-bd97b\" (UniqueName: \"kubernetes.io/projected/11fa077e-79c2-4ffd-9320-168dd602b772-kube-api-access-bd97b\") pod \"11fa077e-79c2-4ffd-9320-168dd602b772\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") "
Oct 30 14:12:33.051968 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.051885    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-xks-ca\") pod \"11fa077e-79c2-4ffd-9320-168dd602b772\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") "
Oct 30 14:12:33.052033 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.052003    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-openshift-ca\") pod \"11fa077e-79c2-4ffd-9320-168dd602b772\" (UID: \"11fa077e-79c2-4ffd-9320-168dd602b772\") "
Oct 30 14:12:33.052409 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.052383    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-openshift-ca" pod "11fa077e-79c2-4ffd-9320-168dd602b772" (UID: "11fa077e-79c2-4ffd-9320-168dd602b772"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:33.054044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.054025    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-xks-ca" pod "11fa077e-79c2-4ffd-9320-168dd602b772" (UID: "11fa077e-79c2-4ffd-9320-168dd602b772"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:33.054135 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.054091    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11fa077e-79c2-4ffd-9320-168dd602b772-kube-api-access-bd97b" pod "11fa077e-79c2-4ffd-9320-168dd602b772" (UID: "11fa077e-79c2-4ffd-9320-168dd602b772"). InnerVolumeSpecName "kube-api-access-bd97b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:33.153813 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.153712    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:33.153813 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.153744    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11fa077e-79c2-4ffd-9320-168dd602b772-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:33.153813 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.153755    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bd97b\" (UniqueName: \"kubernetes.io/projected/11fa077e-79c2-4ffd-9320-168dd602b772-kube-api-access-bd97b\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:33.866104 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.866066    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc" event={"ID":"11fa077e-79c2-4ffd-9320-168dd602b772","Type":"ContainerDied","Data":"f34b8dd3aa26933b7e2a021f33371649b6c957225c09cf305a5e1a9d3cacb463"}
Oct 30 14:12:33.866433 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.866398    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f34b8dd3aa26933b7e2a021f33371649b6c957225c09cf305a5e1a9d3cacb463"
Oct 30 14:12:33.866670 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.866654    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84fda1aa-bmqbc"
Oct 30 14:12:33.878072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.878046    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="604135f2-4b64-4c25-980c-df8d4315a28e" path="/var/lib/kubelet/pods/604135f2-4b64-4c25-980c-df8d4315a28e/volumes"
Oct 30 14:12:33.878492 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.878478    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5606a9d-5078-4203-a318-0ae43169f13b" path="/var/lib/kubelet/pods/b5606a9d-5078-4203-a318-0ae43169f13b/volumes"
Oct 30 14:12:33.878888 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:33.878873    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbb3f7ed-f161-49d2-8654-a4f45e615352" path="/var/lib/kubelet/pods/cbb3f7ed-f161-49d2-8654-a4f45e615352/volumes"
Oct 30 14:12:35.422262 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:35.422219    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"]
Oct 30 14:12:35.422812 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:35.422477    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy" containerID="cri-o://287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e" gracePeriod=30
Oct 30 14:12:35.526473 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:35.526392    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" podUID="6e231fd7-3743-4341-83ba-61583a6461fb" containerName="payload-processing" containerID="cri-o://61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8" gracePeriod=25
Oct 30 14:12:35.553013 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:35.552957    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" podUID="cd9eb694-130e-4dcd-a814-9487970b78d2" containerName="payload-pre-processing" containerID="cri-o://3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe" gracePeriod=25
Oct 30 14:12:37.213087 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:37.213045    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:12:37.654799 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:37.654756    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" podUID="cd9eb694-130e-4dcd-a814-9487970b78d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:37.654980 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:37.654856    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" podUID="6e231fd7-3743-4341-83ba-61583a6461fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:38.272610 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:38.270683    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:12:38.706719 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:38.706679    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"]
Oct 30 14:12:38.717964 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:38.717934    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-29a3f3-8wjht"]
Oct 30 14:12:39.878086 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:39.878056    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccbef97-06be-42b9-9edd-36db0c2922e9" path="/var/lib/kubelet/pods/4ccbef97-06be-42b9-9edd-36db0c2922e9/volumes"
Oct 30 14:12:40.685441 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.685414    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:12:40.737461 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.737367    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-podinfo\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.737461 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.737429    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-data\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.737763 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.737712    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istiod-ca-cert\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.737845    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-credential-socket\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.737860    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-data" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.737954    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-socket\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738038    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-kube-api-access-szrhx\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738065    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-token\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-token\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738123    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-certs\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738167    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-envoy\") pod \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\" (UID: \"59c27364-1fb1-41b1-aee1-5491e06b4f6f\") "
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738189    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-credential-socket" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738129    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istiod-ca-cert" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738433    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738444    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.738592 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738453    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.739384 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.738689    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-certs" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:40.739834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.739810    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-socket" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:40.742204 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.742159    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-kube-api-access-szrhx" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "kube-api-access-szrhx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:40.742204 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.742158    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-podinfo" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:40.742436 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.742410    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-envoy" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:40.742525 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.742501    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-token" pod "59c27364-1fb1-41b1-aee1-5491e06b4f6f" (UID: "59c27364-1fb1-41b1-aee1-5491e06b4f6f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:40.840036 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.839997    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.840036 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.840031    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.840036 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.840044    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.840294 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.840055    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59c27364-1fb1-41b1-aee1-5491e06b4f6f-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.840294 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.840069    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szrhx\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-kube-api-access-szrhx\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.840294 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.840082    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59c27364-1fb1-41b1-aee1-5491e06b4f6f-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:40.921189 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.921153    4291 generic.go:356] "Generic (PLEG): container finished" podID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerID="287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e" exitCode=0
Oct 30 14:12:40.921683 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.921207    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"
Oct 30 14:12:40.921683 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.921239    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" event={"ID":"59c27364-1fb1-41b1-aee1-5491e06b4f6f","Type":"ContainerDied","Data":"287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e"}
Oct 30 14:12:40.921683 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.921292    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c" event={"ID":"59c27364-1fb1-41b1-aee1-5491e06b4f6f","Type":"ContainerDied","Data":"1cef2097d973ded8456621d83a4c204d8155c46ad8b6e11c57263a273d17b071"}
Oct 30 14:12:40.921683 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.921314    4291 scope.go:122] "RemoveContainer" containerID="287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e"
Oct 30 14:12:40.933014 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.932995    4291 scope.go:122] "RemoveContainer" containerID="287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e"
Oct 30 14:12:40.933280 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:40.933261    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e\": container with ID starting with 287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e not found: ID does not exist" containerID="287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e"
Oct 30 14:12:40.933318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.933290    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e"} err="failed to get container status \"287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e\": rpc error: code = NotFound desc = could not find container \"287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e\": container with ID starting with 287ab98c5e732e4cf8f6cc6c8f138127e459249b4171acc897d5a834c557ce2e not found: ID does not exist"
Oct 30 14:12:40.971711 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.971677    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"]
Oct 30 14:12:40.974700 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:40.974679    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-67bf986f99-qwt8c"]
Oct 30 14:12:41.877538 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:41.877503    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" path="/var/lib/kubelet/pods/59c27364-1fb1-41b1-aee1-5491e06b4f6f/volumes"
Oct 30 14:12:43.031578 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:43.031542    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"]
Oct 30 14:12:43.043028 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:43.042997    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-17297c37-rsdsg"]
Oct 30 14:12:43.907630 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:43.907589    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10dbb17d-fc1b-4d4f-9908-62cb19a7fa41" path="/var/lib/kubelet/pods/10dbb17d-fc1b-4d4f-9908-62cb19a7fa41/volumes"
Oct 30 14:12:44.089704 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:44.089661    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"]
Oct 30 14:12:44.101157 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:44.101124    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-c684d2-5h2xn"]
Oct 30 14:12:45.877921 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:45.877889    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" path="/var/lib/kubelet/pods/cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3/volumes"
Oct 30 14:12:46.093580 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:46.093546    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"]
Oct 30 14:12:46.112624 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:46.105049    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-f9ebb5-g4g7q"]
Oct 30 14:12:47.464531 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:47.464493    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"]
Oct 30 14:12:47.465098 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:47.464763    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy" containerID="cri-o://7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69" gracePeriod=30
Oct 30 14:12:47.654202 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:47.654166    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" podUID="6e231fd7-3743-4341-83ba-61583a6461fb" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:47.654407 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:47.654230    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" podUID="cd9eb694-130e-4dcd-a814-9487970b78d2" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 14:12:47.877653 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:47.877594    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2a00d38-316c-4bdd-a105-b55ca75a40ae" path="/var/lib/kubelet/pods/d2a00d38-316c-4bdd-a105-b55ca75a40ae/volumes"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.222162    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"]
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223150    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" containerName="revoke-keys"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223174    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" containerName="revoke-keys"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223187    4291 cpu_manager.go:414] "containerMap: removing container" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223204    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223232    4291 cpu_manager.go:414] "containerMap: removing container" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223241    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223266    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223274    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223300    4291 cpu_manager.go:414] "containerMap: removing container" podUID="11fa077e-79c2-4ffd-9320-168dd602b772" containerName="revoke-keys"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223307    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11fa077e-79c2-4ffd-9320-168dd602b772" containerName="revoke-keys"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223320    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cbb3f7ed-f161-49d2-8654-a4f45e615352" containerName="payload-pre-processing"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223327    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbb3f7ed-f161-49d2-8654-a4f45e615352" containerName="payload-pre-processing"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223366    4291 cpu_manager.go:414] "containerMap: removing container" podUID="604135f2-4b64-4c25-980c-df8d4315a28e" containerName="payload-processing"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223373    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="604135f2-4b64-4c25-980c-df8d4315a28e" containerName="payload-processing"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223396    4291 cpu_manager.go:414] "containerMap: removing container" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223403    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223607    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf4f9e07-f3b4-4cee-a7d7-7674cd961bc3" containerName="revoke-keys"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223626    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fe0cce-0dda-410b-bf07-5c5db90c52b8" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223663    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="59c27364-1fb1-41b1-aee1-5491e06b4f6f" containerName="istio-proxy"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223674    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8c75286-fcfd-4d26-9e75-de16b42a094c" containerName="main"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223691    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="11fa077e-79c2-4ffd-9320-168dd602b772" containerName="revoke-keys"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223700    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbb3f7ed-f161-49d2-8654-a4f45e615352" containerName="payload-pre-processing"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223719    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="604135f2-4b64-4c25-980c-df8d4315a28e" containerName="payload-processing"
Oct 30 14:12:48.225857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.223729    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b145b65-81da-48b6-afd6-5ad1ddef2572" containerName="main"
Oct 30 14:12:48.228908 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.228874    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.241568 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.241542    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"]
Oct 30 14:12:48.243496 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.243476    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:12:48.243692 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.243678    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:12:48.246698 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.246675    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:48.250353 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.250334    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:12:48.349451 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.349415    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.349659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.349473    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.349744 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.349681    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbzzj\" (UniqueName: \"kubernetes.io/projected/716d3083-3b76-4af3-8a0c-49b876f78c6f-kube-api-access-vbzzj\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.451120 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.451071    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbzzj\" (UniqueName: \"kubernetes.io/projected/716d3083-3b76-4af3-8a0c-49b876f78c6f-kube-api-access-vbzzj\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.451330 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.451158    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.451330 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.451194    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.451992 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.451971    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.453721 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.453698    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.470285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.470255    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbzzj\" (UniqueName: \"kubernetes.io/projected/716d3083-3b76-4af3-8a0c-49b876f78c6f-kube-api-access-vbzzj\") pod \"maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.546012 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.545936    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:48.845781 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.845757    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"]
Oct 30 14:12:48.847728 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:12:48.847698    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod716d3083_3b76_4af3_8a0c_49b876f78c6f.slice/crio-f74749161537eef8ecfedb173a4707df0f9c45ee5bd6c564b25c4d1bd028141a WatchSource:0}: Error finding container f74749161537eef8ecfedb173a4707df0f9c45ee5bd6c564b25c4d1bd028141a: Status 404 returned error can't find the container with id f74749161537eef8ecfedb173a4707df0f9c45ee5bd6c564b25c4d1bd028141a
Oct 30 14:12:48.978981 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:48.978949    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl" event={"ID":"716d3083-3b76-4af3-8a0c-49b876f78c6f","Type":"ContainerStarted","Data":"f74749161537eef8ecfedb173a4707df0f9c45ee5bd6c564b25c4d1bd028141a"}
Oct 30 14:12:49.987367 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:49.987266    4291 generic.go:356] "Generic (PLEG): container finished" podID="716d3083-3b76-4af3-8a0c-49b876f78c6f" containerID="81389063a41f0825318564227fbe8101a8756cba62302d82d8f40fd554f68cf9" exitCode=0
Oct 30 14:12:49.987777 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:49.987358    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl" event={"ID":"716d3083-3b76-4af3-8a0c-49b876f78c6f","Type":"ContainerDied","Data":"81389063a41f0825318564227fbe8101a8756cba62302d82d8f40fd554f68cf9"}
Oct 30 14:12:51.138329 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.138304    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:51.280925 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.280834    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-openshift-ca\") pod \"716d3083-3b76-4af3-8a0c-49b876f78c6f\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") "
Oct 30 14:12:51.281098 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.281002    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-xks-ca\") pod \"716d3083-3b76-4af3-8a0c-49b876f78c6f\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") "
Oct 30 14:12:51.281098 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.281036    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/716d3083-3b76-4af3-8a0c-49b876f78c6f-kube-api-access-vbzzj\" (UniqueName: \"kubernetes.io/projected/716d3083-3b76-4af3-8a0c-49b876f78c6f-kube-api-access-vbzzj\") pod \"716d3083-3b76-4af3-8a0c-49b876f78c6f\" (UID: \"716d3083-3b76-4af3-8a0c-49b876f78c6f\") "
Oct 30 14:12:51.281239 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.281214    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-openshift-ca" pod "716d3083-3b76-4af3-8a0c-49b876f78c6f" (UID: "716d3083-3b76-4af3-8a0c-49b876f78c6f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:51.281390 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.281370    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:51.283087 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.283064    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-xks-ca" pod "716d3083-3b76-4af3-8a0c-49b876f78c6f" (UID: "716d3083-3b76-4af3-8a0c-49b876f78c6f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:51.283220 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.283205    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/716d3083-3b76-4af3-8a0c-49b876f78c6f-kube-api-access-vbzzj" pod "716d3083-3b76-4af3-8a0c-49b876f78c6f" (UID: "716d3083-3b76-4af3-8a0c-49b876f78c6f"). InnerVolumeSpecName "kube-api-access-vbzzj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:51.382529 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.382494    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/716d3083-3b76-4af3-8a0c-49b876f78c6f-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:51.382529 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:51.382525    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbzzj\" (UniqueName: \"kubernetes.io/projected/716d3083-3b76-4af3-8a0c-49b876f78c6f-kube-api-access-vbzzj\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.004877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.004844    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"
Oct 30 14:12:52.004877 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.004872    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl" event={"ID":"716d3083-3b76-4af3-8a0c-49b876f78c6f","Type":"ContainerDied","Data":"f74749161537eef8ecfedb173a4707df0f9c45ee5bd6c564b25c4d1bd028141a"}
Oct 30 14:12:52.005166 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.004920    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f74749161537eef8ecfedb173a4707df0f9c45ee5bd6c564b25c4d1bd028141a"
Oct 30 14:12:52.721286 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.721261    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 14:12:52.903991 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.903949    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-data\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.903991 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904001    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-envoy\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904039    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-socket\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904114    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-certs\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904161    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-credential-socket\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904205    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3397ccd9-392f-4245-a839-48fd4da8d393-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3397ccd9-392f-4245-a839-48fd4da8d393-istiod-ca-cert\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904223    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-data" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.904252 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904238    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-kube-api-access-hkdms\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-kube-api-access-hkdms\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904296    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3397ccd9-392f-4245-a839-48fd4da8d393-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3397ccd9-392f-4245-a839-48fd4da8d393-istio-podinfo\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904329    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-istio-token\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-istio-token\") pod \"3397ccd9-392f-4245-a839-48fd4da8d393\" (UID: \"3397ccd9-392f-4245-a839-48fd4da8d393\") "
Oct 30 14:12:52.904595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904461    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-socket" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.904595 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904479    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-certs" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.904828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904731    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3397ccd9-392f-4245-a839-48fd4da8d393-istiod-ca-cert" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:52.904828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904777    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-credential-socket" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.904945 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904863    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3397ccd9-392f-4245-a839-48fd4da8d393-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.904945 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904880    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.904945 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904888    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.904945 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.904897    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:52.906315 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.906293    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-envoy" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:52.906491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.906475    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-kube-api-access-hkdms" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "kube-api-access-hkdms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:52.906859 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.906842    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3397ccd9-392f-4245-a839-48fd4da8d393-istio-podinfo" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:52.906940 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:52.906915    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-istio-token" pod "3397ccd9-392f-4245-a839-48fd4da8d393" (UID: "3397ccd9-392f-4245-a839-48fd4da8d393"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:53.005915 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.005878    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.005915 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.005913    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hkdms\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-kube-api-access-hkdms\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.006176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.005928    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3397ccd9-392f-4245-a839-48fd4da8d393-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.006176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.005942    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3397ccd9-392f-4245-a839-48fd4da8d393-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.006176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.005953    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3397ccd9-392f-4245-a839-48fd4da8d393-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:53.017813 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.017756    4291 generic.go:356] "Generic (PLEG): container finished" podID="3397ccd9-392f-4245-a839-48fd4da8d393" containerID="7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69" exitCode=0
Oct 30 14:12:53.018067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.017898    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" event={"ID":"3397ccd9-392f-4245-a839-48fd4da8d393","Type":"ContainerDied","Data":"7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69"}
Oct 30 14:12:53.018067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.018000    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn" event={"ID":"3397ccd9-392f-4245-a839-48fd4da8d393","Type":"ContainerDied","Data":"4ba8388ca43f065537145ccb908070fa48b57ed70a068cd4f5d20ea479bb6d5c"}
Oct 30 14:12:53.018067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.018039    4291 scope.go:122] "RemoveContainer" containerID="7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69"
Oct 30 14:12:53.021645 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.018262    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"
Oct 30 14:12:53.031693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.031657    4291 scope.go:122] "RemoveContainer" containerID="33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1"
Oct 30 14:12:53.042238 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.042210    4291 scope.go:122] "RemoveContainer" containerID="7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69"
Oct 30 14:12:53.042552 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:53.042525    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69\": container with ID starting with 7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69 not found: ID does not exist" containerID="7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69"
Oct 30 14:12:53.042618 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.042555    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69"} err="failed to get container status \"7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69\": rpc error: code = NotFound desc = could not find container \"7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69\": container with ID starting with 7591f413491194c1a8437eb541b6f2eb51e7e0ccca6b6d4d9236915c82fd9f69 not found: ID does not exist"
Oct 30 14:12:53.042618 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.042580    4291 scope.go:122] "RemoveContainer" containerID="33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1"
Oct 30 14:12:53.042907 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:53.042888    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1\": container with ID starting with 33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1 not found: ID does not exist" containerID="33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1"
Oct 30 14:12:53.042972 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.042915    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1"} err="failed to get container status \"33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1\": rpc error: code = NotFound desc = could not find container \"33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1\": container with ID starting with 33dd4b94e2af92c5c4dd1b92fc99cf795c96521ae33c90a938fc1a55dba6f2e1 not found: ID does not exist"
Oct 30 14:12:53.046288 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.046266    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"]
Oct 30 14:12:53.078464 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.078424    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"]
Oct 30 14:12:53.081224 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.081195    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"]
Oct 30 14:12:53.098822 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.098784    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-29a3f3-openshift-default-775b8b6c9f-cmjmn"]
Oct 30 14:12:53.147395 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.147359    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:12:53.519321 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.519285    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"]
Oct 30 14:12:53.519529 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.519504    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy" containerID="cri-o://a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38" gracePeriod=30
Oct 30 14:12:53.877067 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:53.877033    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" path="/var/lib/kubelet/pods/3397ccd9-392f-4245-a839-48fd4da8d393/volumes"
Oct 30 14:12:54.037569 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.037539    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"]
Oct 30 14:12:54.039981 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.039960    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-33f32700-t9nlf"]
Oct 30 14:12:54.084168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.084133    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"]
Oct 30 14:12:54.084608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.084584    4291 cpu_manager.go:414] "containerMap: removing container" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy"
Oct 30 14:12:54.084608 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.084602    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy"
Oct 30 14:12:54.084779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.084618    4291 cpu_manager.go:414] "containerMap: removing container" podUID="716d3083-3b76-4af3-8a0c-49b876f78c6f" containerName="revoke-keys"
Oct 30 14:12:54.084779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.084624    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="716d3083-3b76-4af3-8a0c-49b876f78c6f" containerName="revoke-keys"
Oct 30 14:12:54.084779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.084719    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy"
Oct 30 14:12:54.084779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.084730    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="716d3083-3b76-4af3-8a0c-49b876f78c6f" containerName="revoke-keys"
Oct 30 14:12:54.089363 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.089346    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.104184 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.104156    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:12:54.104347 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.104226    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:12:54.106726 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.106702    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"]
Oct 30 14:12:54.169438 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.169354    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:12:54.169598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.169438    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:12:54.217982 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.217947    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46mml\" (UniqueName: \"kubernetes.io/projected/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-kube-api-access-46mml\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.218180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.218009    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.218180 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.218108    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.319019 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.318979    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46mml\" (UniqueName: \"kubernetes.io/projected/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-kube-api-access-46mml\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.319216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.319049    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.319216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.319090    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.319751 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.319724    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.321552 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.321522    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.342665 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.342620    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46mml\" (UniqueName: \"kubernetes.io/projected/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-kube-api-access-46mml\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.402540 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.402497    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:54.711821 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:54.711787    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"]
Oct 30 14:12:54.713973 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:12:54.713947    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf91a4e75_a0f0_4cfd_9f3d_0f8c3dc046ae.slice/crio-3063656ff08ae26c4eb5ab25c3ef7849709d79f2747eec8565631a331f76ba40 WatchSource:0}: Error finding container 3063656ff08ae26c4eb5ab25c3ef7849709d79f2747eec8565631a331f76ba40: Status 404 returned error can't find the container with id 3063656ff08ae26c4eb5ab25c3ef7849709d79f2747eec8565631a331f76ba40
Oct 30 14:12:55.034205 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.034167    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd" event={"ID":"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae","Type":"ContainerStarted","Data":"0c533cf5558cc833a083fbf99cb93899b32bcffdb8502541415cd6d28d0a9afc"}
Oct 30 14:12:55.034601 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.034232    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd" event={"ID":"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae","Type":"ContainerStarted","Data":"3063656ff08ae26c4eb5ab25c3ef7849709d79f2747eec8565631a331f76ba40"}
Oct 30 14:12:55.072929 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.072868    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd" podStartSLOduration=1.072851323 podStartE2EDuration="1.072851323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:12:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:12:55.070914299 +0000 UTC m=+4185.331152533" watchObservedRunningTime="2026-10-30 14:12:55.072851323 +0000 UTC m=+4185.333089557"
Oct 30 14:12:55.807381 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.807347    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 14:12:55.829150 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.829123    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 14:12:55.838615 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.838590    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd9eb694-130e-4dcd-a814-9487970b78d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb694-130e-4dcd-a814-9487970b78d2-plugins-config-volume\") pod \"cd9eb694-130e-4dcd-a814-9487970b78d2\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") "
Oct 30 14:12:55.838764 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.838735    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e231fd7-3743-4341-83ba-61583a6461fb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e231fd7-3743-4341-83ba-61583a6461fb-plugins-config-volume\") pod \"6e231fd7-3743-4341-83ba-61583a6461fb\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") "
Oct 30 14:12:55.838811 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.838791    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd9eb694-130e-4dcd-a814-9487970b78d2-kube-api-access-s6xxh\" (UniqueName: \"kubernetes.io/projected/cd9eb694-130e-4dcd-a814-9487970b78d2-kube-api-access-s6xxh\") pod \"cd9eb694-130e-4dcd-a814-9487970b78d2\" (UID: \"cd9eb694-130e-4dcd-a814-9487970b78d2\") "
Oct 30 14:12:55.838857 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.838819    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e231fd7-3743-4341-83ba-61583a6461fb-kube-api-access-269ms\" (UniqueName: \"kubernetes.io/projected/6e231fd7-3743-4341-83ba-61583a6461fb-kube-api-access-269ms\") pod \"6e231fd7-3743-4341-83ba-61583a6461fb\" (UID: \"6e231fd7-3743-4341-83ba-61583a6461fb\") "
Oct 30 14:12:55.839132 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.839108    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd9eb694-130e-4dcd-a814-9487970b78d2-plugins-config-volume" pod "cd9eb694-130e-4dcd-a814-9487970b78d2" (UID: "cd9eb694-130e-4dcd-a814-9487970b78d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:55.839177 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.839139    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e231fd7-3743-4341-83ba-61583a6461fb-plugins-config-volume" pod "6e231fd7-3743-4341-83ba-61583a6461fb" (UID: "6e231fd7-3743-4341-83ba-61583a6461fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:55.840920 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.840893    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e231fd7-3743-4341-83ba-61583a6461fb-kube-api-access-269ms" pod "6e231fd7-3743-4341-83ba-61583a6461fb" (UID: "6e231fd7-3743-4341-83ba-61583a6461fb"). InnerVolumeSpecName "kube-api-access-269ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:55.841025 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.840944    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd9eb694-130e-4dcd-a814-9487970b78d2-kube-api-access-s6xxh" pod "cd9eb694-130e-4dcd-a814-9487970b78d2" (UID: "cd9eb694-130e-4dcd-a814-9487970b78d2"). InnerVolumeSpecName "kube-api-access-s6xxh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:55.877875 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.877836    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf8c180e-362e-483b-9fca-edcf1c593ea5" path="/var/lib/kubelet/pods/bf8c180e-362e-483b-9fca-edcf1c593ea5/volumes"
Oct 30 14:12:55.939843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.939792    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e231fd7-3743-4341-83ba-61583a6461fb-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:55.939843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.939835    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6xxh\" (UniqueName: \"kubernetes.io/projected/cd9eb694-130e-4dcd-a814-9487970b78d2-kube-api-access-s6xxh\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:55.939843 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.939847    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-269ms\" (UniqueName: \"kubernetes.io/projected/6e231fd7-3743-4341-83ba-61583a6461fb-kube-api-access-269ms\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:55.940106 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:55.939860    4291 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd9eb694-130e-4dcd-a814-9487970b78d2-plugins-config-volume\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:56.042183 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.042075    4291 generic.go:356] "Generic (PLEG): container finished" podID="6e231fd7-3743-4341-83ba-61583a6461fb" containerID="61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8" exitCode=0
Oct 30 14:12:56.042183 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.042123    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"
Oct 30 14:12:56.042183 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.042163    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" event={"ID":"6e231fd7-3743-4341-83ba-61583a6461fb","Type":"ContainerDied","Data":"61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8"}
Oct 30 14:12:56.042814 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.042217    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz" event={"ID":"6e231fd7-3743-4341-83ba-61583a6461fb","Type":"ContainerDied","Data":"488cac855ed29603d85ad3e6d16380ba90ea64a2de4ed14df052cb367416eeb1"}
Oct 30 14:12:56.042814 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.042243    4291 scope.go:122] "RemoveContainer" containerID="61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8"
Oct 30 14:12:56.043874 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.043842    4291 generic.go:356] "Generic (PLEG): container finished" podID="cd9eb694-130e-4dcd-a814-9487970b78d2" containerID="3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe" exitCode=0
Oct 30 14:12:56.043976 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.043889    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"
Oct 30 14:12:56.043976 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.043922    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" event={"ID":"cd9eb694-130e-4dcd-a814-9487970b78d2","Type":"ContainerDied","Data":"3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe"}
Oct 30 14:12:56.044121 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.044008    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv" event={"ID":"cd9eb694-130e-4dcd-a814-9487970b78d2","Type":"ContainerDied","Data":"dad1ea4c19a34b970225f8f02706dc58dbb437969776a716deb0fbcb38f301c0"}
Oct 30 14:12:56.045832 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.045808    4291 generic.go:356] "Generic (PLEG): container finished" podID="f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" containerID="0c533cf5558cc833a083fbf99cb93899b32bcffdb8502541415cd6d28d0a9afc" exitCode=0
Oct 30 14:12:56.045949 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.045902    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd" event={"ID":"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae","Type":"ContainerDied","Data":"0c533cf5558cc833a083fbf99cb93899b32bcffdb8502541415cd6d28d0a9afc"}
Oct 30 14:12:56.053478 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.053458    4291 scope.go:122] "RemoveContainer" containerID="61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8"
Oct 30 14:12:56.053794 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:56.053765    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8\": container with ID starting with 61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8 not found: ID does not exist" containerID="61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8"
Oct 30 14:12:56.053860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.053797    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8"} err="failed to get container status \"61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8\": rpc error: code = NotFound desc = could not find container \"61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8\": container with ID starting with 61a4cd3a1431daeba4878df5e9833db303a870ec368a9f5585a070ade4e374e8 not found: ID does not exist"
Oct 30 14:12:56.053860 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.053813    4291 scope.go:122] "RemoveContainer" containerID="3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe"
Oct 30 14:12:56.063229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.063207    4291 scope.go:122] "RemoveContainer" containerID="3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe"
Oct 30 14:12:56.063543 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:56.063519    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe\": container with ID starting with 3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe not found: ID does not exist" containerID="3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe"
Oct 30 14:12:56.063591 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.063551    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe"} err="failed to get container status \"3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe\": rpc error: code = NotFound desc = could not find container \"3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe\": container with ID starting with 3e7aba1ca6a4e6d1da238f524ab7557c648f0c4a467b25c511fe26665af1ccbe not found: ID does not exist"
Oct 30 14:12:56.134770 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.134736    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"]
Oct 30 14:12:56.137503 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.137479    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-29a3f3-c9d8c6d84-6n9jz"]
Oct 30 14:12:56.170508 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.170475    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"]
Oct 30 14:12:56.172883 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.172859    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-29a3f3-78dbf5d59f-mkpsv"]
Oct 30 14:12:56.250435 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.250402    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"]
Oct 30 14:12:56.250872 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:56.250839    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy" containerID="cri-o://61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d" gracePeriod=30
Oct 30 14:12:57.201318 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.201292    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:57.253423 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.253386    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-openshift-ca\") pod \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") "
Oct 30 14:12:57.253423 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.253426    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-xks-ca\") pod \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") "
Oct 30 14:12:57.253706 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.253532    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-kube-api-access-46mml\" (UniqueName: \"kubernetes.io/projected/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-kube-api-access-46mml\") pod \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\" (UID: \"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae\") "
Oct 30 14:12:57.253834 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.253805    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-openshift-ca" pod "f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" (UID: "f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:57.255602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.255574    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-xks-ca" pod "f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" (UID: "f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:12:57.255726 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.255618    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-kube-api-access-46mml" pod "f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" (UID: "f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae"). InnerVolumeSpecName "kube-api-access-46mml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:57.354882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.354844    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:57.354882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.354875    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:57.354882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.354891    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46mml\" (UniqueName: \"kubernetes.io/projected/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae-kube-api-access-46mml\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:57.877693 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.877657    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e231fd7-3743-4341-83ba-61583a6461fb" path="/var/lib/kubelet/pods/6e231fd7-3743-4341-83ba-61583a6461fb/volumes"
Oct 30 14:12:57.878158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:57.878137    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd9eb694-130e-4dcd-a814-9487970b78d2" path="/var/lib/kubelet/pods/cd9eb694-130e-4dcd-a814-9487970b78d2/volumes"
Oct 30 14:12:58.062356 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.062329    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"
Oct 30 14:12:58.062550 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.062367    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd" event={"ID":"f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae","Type":"ContainerDied","Data":"3063656ff08ae26c4eb5ab25c3ef7849709d79f2747eec8565631a331f76ba40"}
Oct 30 14:12:58.062550 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.062417    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3063656ff08ae26c4eb5ab25c3ef7849709d79f2747eec8565631a331f76ba40"
Oct 30 14:12:58.797209 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.797147    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871159    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-kube-api-access-gx77m\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-kube-api-access-gx77m\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871209    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-certs\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871260    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-envoy\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871294    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-podinfo\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871426    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-credential-socket\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871461    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfcce731-6465-4cf0-9b99-7e3748a6818a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcce731-6465-4cf0-9b99-7e3748a6818a-istiod-ca-cert\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871523    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-token\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-token\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871575    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-data\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.871609    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-socket\") pod \"cfcce731-6465-4cf0-9b99-7e3748a6818a\" (UID: \"cfcce731-6465-4cf0-9b99-7e3748a6818a\") "
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.872340    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-certs" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.873442    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfcce731-6465-4cf0-9b99-7e3748a6818a-istiod-ca-cert" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.873734    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-credential-socket" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.874575    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-data" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.874708    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-socket" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.876699 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.876663    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-kube-api-access-gx77m" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "kube-api-access-gx77m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.877603 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.877567    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-podinfo" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:12:58.877603 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.877575    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-envoy" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:12:58.882113 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.882091    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-token" pod "cfcce731-6465-4cf0-9b99-7e3748a6818a" (UID: "cfcce731-6465-4cf0-9b99-7e3748a6818a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:12:58.972720 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972680    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972720 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972721    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cfcce731-6465-4cf0-9b99-7e3748a6818a-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972736    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972753    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972768    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972779    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gx77m\" (UniqueName: \"kubernetes.io/projected/cfcce731-6465-4cf0-9b99-7e3748a6818a-kube-api-access-gx77m\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972796    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972811    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:58.972989 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:58.972826    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cfcce731-6465-4cf0-9b99-7e3748a6818a-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:12:59.070213 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.070121    4291 generic.go:356] "Generic (PLEG): container finished" podID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerID="a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38" exitCode=0
Oct 30 14:12:59.070213 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.070177    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"
Oct 30 14:12:59.070417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.070209    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" event={"ID":"cfcce731-6465-4cf0-9b99-7e3748a6818a","Type":"ContainerDied","Data":"a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38"}
Oct 30 14:12:59.070417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.070269    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv" event={"ID":"cfcce731-6465-4cf0-9b99-7e3748a6818a","Type":"ContainerDied","Data":"0d12a906bdadd431681791011706cbe82142888206363603248ba138965903e8"}
Oct 30 14:12:59.070417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.070290    4291 scope.go:122] "RemoveContainer" containerID="a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38"
Oct 30 14:12:59.081476 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.081455    4291 scope.go:122] "RemoveContainer" containerID="a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38"
Oct 30 14:12:59.081786 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:12:59.081767    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38\": container with ID starting with a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38 not found: ID does not exist" containerID="a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38"
Oct 30 14:12:59.081871 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.081796    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38"} err="failed to get container status \"a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38\": rpc error: code = NotFound desc = could not find container \"a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38\": container with ID starting with a8a013ae87696a32bc6958326cbfbd3bb030a3a143e7d68d769ce6a4cb542c38 not found: ID does not exist"
Oct 30 14:12:59.141465 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.141430    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"]
Oct 30 14:12:59.143895 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.143873    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-c684d2-openshift-default-f55d4676f-9dcsv"]
Oct 30 14:12:59.878522 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:12:59.878487    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" path="/var/lib/kubelet/pods/cfcce731-6465-4cf0-9b99-7e3748a6818a/volumes"
Oct 30 14:13:01.392504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392464    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"]
Oct 30 14:13:01.392905 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392894    4291 cpu_manager.go:414] "containerMap: removing container" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy"
Oct 30 14:13:01.392954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392909    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy"
Oct 30 14:13:01.392954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392935    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy"
Oct 30 14:13:01.392954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392942    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy"
Oct 30 14:13:01.392954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392950    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" containerName="revoke-keys"
Oct 30 14:13:01.392954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392955    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" containerName="revoke-keys"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392963    4291 cpu_manager.go:414] "containerMap: removing container" podUID="cd9eb694-130e-4dcd-a814-9487970b78d2" containerName="payload-pre-processing"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392967    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd9eb694-130e-4dcd-a814-9487970b78d2" containerName="payload-pre-processing"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392974    4291 cpu_manager.go:414] "containerMap: removing container" podUID="6e231fd7-3743-4341-83ba-61583a6461fb" containerName="payload-processing"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.392979    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e231fd7-3743-4341-83ba-61583a6461fb" containerName="payload-processing"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.393048    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="3397ccd9-392f-4245-a839-48fd4da8d393" containerName="istio-proxy"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.393058    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfcce731-6465-4cf0-9b99-7e3748a6818a" containerName="istio-proxy"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.393065    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" containerName="revoke-keys"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.393071    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd9eb694-130e-4dcd-a814-9487970b78d2" containerName="payload-pre-processing"
Oct 30 14:13:01.393123 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.393077    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e231fd7-3743-4341-83ba-61583a6461fb" containerName="payload-processing"
Oct 30 14:13:01.397491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.397464    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.413302 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.413263    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:13:01.415926 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.415888    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"]
Oct 30 14:13:01.419269 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.419241    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"]
Oct 30 14:13:01.423625 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.423601    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.425062 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.425042    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:13:01.425173 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.425071    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:13:01.425242 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.425074    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:13:01.456733 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.456698    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"]
Oct 30 14:13:01.500442 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.500407    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.500660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.500471    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqb9z\" (UniqueName: \"kubernetes.io/projected/7aa44683-5f3b-4df9-94aa-08a7597b4a83-kube-api-access-mqb9z\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.500660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.500494    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqb4q\" (UniqueName: \"kubernetes.io/projected/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-kube-api-access-jqb4q\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.500660 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.500580    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.500859 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.500708    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.500859 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.500787    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.533738 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.533672    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 14:13:01.609076 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609029    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-kube-api-access-5vcmz\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-kube-api-access-5vcmz\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609264 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609095    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-certs\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609264 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609144    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-data\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609264 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609178    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-credential-socket\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609462 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609289    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-envoy\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609462 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609366    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-socket\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609462 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609395    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-token\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-token\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609462 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609427    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de7d1006-839a-487b-89d2-5155bac0c5f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7d1006-839a-487b-89d2-5155bac0c5f1-istiod-ca-cert\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609488    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-podinfo\") pod \"de7d1006-839a-487b-89d2-5155bac0c5f1\" (UID: \"de7d1006-839a-487b-89d2-5155bac0c5f1\") "
Oct 30 14:13:01.609710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609660    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-credential-socket" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:01.609710 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609699    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqb9z\" (UniqueName: \"kubernetes.io/projected/7aa44683-5f3b-4df9-94aa-08a7597b4a83-kube-api-access-mqb9z\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.609876 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609739    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqb4q\" (UniqueName: \"kubernetes.io/projected/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-kube-api-access-jqb4q\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.609876 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609774    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.609876 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609841    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-certs" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:01.609876 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609853    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.610108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609912    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.610108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609978    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.610108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.609993    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-data" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:01.610108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.610040    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.610108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.610055    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.611384 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.611357    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.612107 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.612085    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.615119 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.615089    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-kube-api-access-5vcmz" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "kube-api-access-5vcmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:01.615237 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.615112    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de7d1006-839a-487b-89d2-5155bac0c5f1-istiod-ca-cert" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:01.615444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.615413    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-socket" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:01.615551 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.615496    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-podinfo" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:01.617355 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.616528    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.617768 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.617746    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.617882 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.617864    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-envoy" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:01.619654 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.619619    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-token" pod "de7d1006-839a-487b-89d2-5155bac0c5f1" (UID: "de7d1006-839a-487b-89d2-5155bac0c5f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:01.631772 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.631741    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqb9z\" (UniqueName: \"kubernetes.io/projected/7aa44683-5f3b-4df9-94aa-08a7597b4a83-kube-api-access-mqb9z\") pod \"maas-api-revoke-sub-5acba7b8-qt8c9\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.632032 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.632003    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqb4q\" (UniqueName: \"kubernetes.io/projected/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-kube-api-access-jqb4q\") pod \"maas-api-revoke-sub-bc477ec3-sjrsv\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:01.711547 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.711507    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.711547 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.711539    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vcmz\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-kube-api-access-5vcmz\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.711547 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.711550    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.711790 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.711560    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.711790 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.711584    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de7d1006-839a-487b-89d2-5155bac0c5f1-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.711790 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.711594    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de7d1006-839a-487b-89d2-5155bac0c5f1-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.711790 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.711604    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de7d1006-839a-487b-89d2-5155bac0c5f1-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:01.716489 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.716458    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:01.752429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:01.752392    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:02.027837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.027809    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"]
Oct 30 14:13:02.030084 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:13:02.030058    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aa44683_5f3b_4df9_94aa_08a7597b4a83.slice/crio-c5b68b033d4d3bfc644eca8b751302905865ad21e7a1033bb287ec08f705adef WatchSource:0}: Error finding container c5b68b033d4d3bfc644eca8b751302905865ad21e7a1033bb287ec08f705adef: Status 404 returned error can't find the container with id c5b68b033d4d3bfc644eca8b751302905865ad21e7a1033bb287ec08f705adef
Oct 30 14:13:02.101022 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.100989    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"]
Oct 30 14:13:02.101934 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:13:02.101900    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9c6ed43_a1f8_48b7_8b92_86cd31b7b607.slice/crio-d53b9e855f9d609459f638cc3637fe6fb78e563414930cca06fa95ec91e73547 WatchSource:0}: Error finding container d53b9e855f9d609459f638cc3637fe6fb78e563414930cca06fa95ec91e73547: Status 404 returned error can't find the container with id d53b9e855f9d609459f638cc3637fe6fb78e563414930cca06fa95ec91e73547
Oct 30 14:13:02.102536 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.102502    4291 generic.go:356] "Generic (PLEG): container finished" podID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerID="61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d" exitCode=0
Oct 30 14:13:02.102681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.102578    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"
Oct 30 14:13:02.102681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.102580    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" event={"ID":"de7d1006-839a-487b-89d2-5155bac0c5f1","Type":"ContainerDied","Data":"61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d"}
Oct 30 14:13:02.102681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.102649    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj" event={"ID":"de7d1006-839a-487b-89d2-5155bac0c5f1","Type":"ContainerDied","Data":"5a95c413bfb2ca1259beefa20b0372701fa56c0fb5dc8a52dc3458d8b2531fae"}
Oct 30 14:13:02.102681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.102681    4291 scope.go:122] "RemoveContainer" containerID="61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d"
Oct 30 14:13:02.104227 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.104194    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9" event={"ID":"7aa44683-5f3b-4df9-94aa-08a7597b4a83","Type":"ContainerStarted","Data":"c5b68b033d4d3bfc644eca8b751302905865ad21e7a1033bb287ec08f705adef"}
Oct 30 14:13:02.117037 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.117015    4291 scope.go:122] "RemoveContainer" containerID="61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d"
Oct 30 14:13:02.117351 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:13:02.117328    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d\": container with ID starting with 61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d not found: ID does not exist" containerID="61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d"
Oct 30 14:13:02.117409 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.117364    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d"} err="failed to get container status \"61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d\": rpc error: code = NotFound desc = could not find container \"61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d\": container with ID starting with 61379aaea169b8cceb973c570dfe0294d9ac5d52e6fdf826b26e5b6a9a885b3d not found: ID does not exist"
Oct 30 14:13:02.153976 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.153941    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"]
Oct 30 14:13:02.156544 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:02.156510    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-f9ebb5-openshift-default-c487699ff-b7tpj"]
Oct 30 14:13:03.112662 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:03.112564    4291 generic.go:356] "Generic (PLEG): container finished" podID="c9c6ed43-a1f8-48b7-8b92-86cd31b7b607" containerID="69e1c133acf880976f69709cb83a71261e7f4f6892a9aba5ea0b82b988e9f673" exitCode=0
Oct 30 14:13:03.113083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:03.112674    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv" event={"ID":"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607","Type":"ContainerDied","Data":"69e1c133acf880976f69709cb83a71261e7f4f6892a9aba5ea0b82b988e9f673"}
Oct 30 14:13:03.113083 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:03.112734    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv" event={"ID":"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607","Type":"ContainerStarted","Data":"d53b9e855f9d609459f638cc3637fe6fb78e563414930cca06fa95ec91e73547"}
Oct 30 14:13:03.115158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:03.115138    4291 generic.go:356] "Generic (PLEG): container finished" podID="7aa44683-5f3b-4df9-94aa-08a7597b4a83" containerID="d41c12d7884ea002135e91a82dd48f184a6b2f9b85a4e95d828a02e577779e78" exitCode=0
Oct 30 14:13:03.115251 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:03.115191    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9" event={"ID":"7aa44683-5f3b-4df9-94aa-08a7597b4a83","Type":"ContainerDied","Data":"d41c12d7884ea002135e91a82dd48f184a6b2f9b85a4e95d828a02e577779e78"}
Oct 30 14:13:03.877615 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:03.877580    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" path="/var/lib/kubelet/pods/de7d1006-839a-487b-89d2-5155bac0c5f1/volumes"
Oct 30 14:13:04.034125 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.034093    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"]
Oct 30 14:13:04.045887 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.045857    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"]
Oct 30 14:13:04.048487 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.048454    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6dc75877-trlrf"]
Oct 30 14:13:04.050506 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.050487    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7390182-wb76j"]
Oct 30 14:13:04.270691 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.270609    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:04.303108 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.303084    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:04.338732 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.338701    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-xks-ca\") pod \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") "
Oct 30 14:13:04.338902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.338791    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-openshift-ca\") pod \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") "
Oct 30 14:13:04.338902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.338842    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aa44683-5f3b-4df9-94aa-08a7597b4a83-kube-api-access-mqb9z\" (UniqueName: \"kubernetes.io/projected/7aa44683-5f3b-4df9-94aa-08a7597b4a83-kube-api-access-mqb9z\") pod \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") "
Oct 30 14:13:04.338902 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.338889    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-openshift-ca\") pod \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") "
Oct 30 14:13:04.339035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.338941    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-xks-ca\") pod \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\" (UID: \"7aa44683-5f3b-4df9-94aa-08a7597b4a83\") "
Oct 30 14:13:04.339035 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.338968    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-kube-api-access-jqb4q\" (UniqueName: \"kubernetes.io/projected/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-kube-api-access-jqb4q\") pod \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\" (UID: \"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607\") "
Oct 30 14:13:04.339257 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.339199    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-openshift-ca" pod "7aa44683-5f3b-4df9-94aa-08a7597b4a83" (UID: "7aa44683-5f3b-4df9-94aa-08a7597b4a83"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:04.339257 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.339242    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-openshift-ca" pod "c9c6ed43-a1f8-48b7-8b92-86cd31b7b607" (UID: "c9c6ed43-a1f8-48b7-8b92-86cd31b7b607"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:04.339413 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.339394    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:04.339481 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.339418    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:04.341255 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.341223    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-xks-ca" pod "c9c6ed43-a1f8-48b7-8b92-86cd31b7b607" (UID: "c9c6ed43-a1f8-48b7-8b92-86cd31b7b607"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:04.341365 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.341314    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-xks-ca" pod "7aa44683-5f3b-4df9-94aa-08a7597b4a83" (UID: "7aa44683-5f3b-4df9-94aa-08a7597b4a83"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:04.341425 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.341386    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-kube-api-access-jqb4q" pod "c9c6ed43-a1f8-48b7-8b92-86cd31b7b607" (UID: "c9c6ed43-a1f8-48b7-8b92-86cd31b7b607"). InnerVolumeSpecName "kube-api-access-jqb4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:04.341425 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.341408    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aa44683-5f3b-4df9-94aa-08a7597b4a83-kube-api-access-mqb9z" pod "7aa44683-5f3b-4df9-94aa-08a7597b4a83" (UID: "7aa44683-5f3b-4df9-94aa-08a7597b4a83"). InnerVolumeSpecName "kube-api-access-mqb9z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:04.440553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.440462    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:04.440553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.440494    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqb9z\" (UniqueName: \"kubernetes.io/projected/7aa44683-5f3b-4df9-94aa-08a7597b4a83-kube-api-access-mqb9z\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:04.440553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.440508    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7aa44683-5f3b-4df9-94aa-08a7597b4a83-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:04.440553 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:04.440522    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqb4q\" (UniqueName: \"kubernetes.io/projected/c9c6ed43-a1f8-48b7-8b92-86cd31b7b607-kube-api-access-jqb4q\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:05.035682 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.032510    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"]
Oct 30 14:13:05.041887 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.041849    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f873afaa-ldv87"]
Oct 30 14:13:05.143008 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.142975    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv"
Oct 30 14:13:05.143199 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.143006    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc477ec3-sjrsv" event={"ID":"c9c6ed43-a1f8-48b7-8b92-86cd31b7b607","Type":"ContainerDied","Data":"d53b9e855f9d609459f638cc3637fe6fb78e563414930cca06fa95ec91e73547"}
Oct 30 14:13:05.143199 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.143072    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d53b9e855f9d609459f638cc3637fe6fb78e563414930cca06fa95ec91e73547"
Oct 30 14:13:05.145430 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.145390    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9" event={"ID":"7aa44683-5f3b-4df9-94aa-08a7597b4a83","Type":"ContainerDied","Data":"c5b68b033d4d3bfc644eca8b751302905865ad21e7a1033bb287ec08f705adef"}
Oct 30 14:13:05.145567 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.145445    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c5b68b033d4d3bfc644eca8b751302905865ad21e7a1033bb287ec08f705adef"
Oct 30 14:13:05.145734 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.145708    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5acba7b8-qt8c9"
Oct 30 14:13:05.877784 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.877750    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="118ff43b-bd6b-4f33-ba72-1590e2d3004c" path="/var/lib/kubelet/pods/118ff43b-bd6b-4f33-ba72-1590e2d3004c/volumes"
Oct 30 14:13:05.878251 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.878235    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19790061-c881-4391-9adf-19927958106a" path="/var/lib/kubelet/pods/19790061-c881-4391-9adf-19927958106a/volumes"
Oct 30 14:13:05.878666 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:05.878652    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd30bd6f-7d43-4aee-83e1-ab992521822b" path="/var/lib/kubelet/pods/fd30bd6f-7d43-4aee-83e1-ab992521822b/volumes"
Oct 30 14:13:08.067491 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.067417    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main" containerID="cri-o://289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723" gracePeriod=45
Oct 30 14:13:08.102230 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.102195    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main" containerID="cri-o://9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23" gracePeriod=45
Oct 30 14:13:08.211658 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:13:08.211608    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:13:08.351236 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.351209    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 14:13:08.381565 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.381534    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 14:13:08.385943 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.385921    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-home\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-home\") pod \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") "
Oct 30 14:13:08.386052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.385971    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-dshm\") pod \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") "
Oct 30 14:13:08.386052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386020    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tmp-dir\") pod \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") "
Oct 30 14:13:08.386052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386042    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-kube-api-access-2b8m4\" (UniqueName: \"kubernetes.io/projected/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-kube-api-access-2b8m4\") pod \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") "
Oct 30 14:13:08.386244 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386156    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tls-certs\" (UniqueName: \"kubernetes.io/secret/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tls-certs\") pod \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") "
Oct 30 14:13:08.386244 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386162    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-home" pod "07d433e1-a2e5-46bf-9dbf-e8e946980a1b" (UID: "07d433e1-a2e5-46bf-9dbf-e8e946980a1b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.386244 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386224    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-model-cache\") pod \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\" (UID: \"07d433e1-a2e5-46bf-9dbf-e8e946980a1b\") "
Oct 30 14:13:08.386432 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386303    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tmp-dir" pod "07d433e1-a2e5-46bf-9dbf-e8e946980a1b" (UID: "07d433e1-a2e5-46bf-9dbf-e8e946980a1b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.386516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386493    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-model-cache" pod "07d433e1-a2e5-46bf-9dbf-e8e946980a1b" (UID: "07d433e1-a2e5-46bf-9dbf-e8e946980a1b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.386627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386561    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.386627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386580    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.386627 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.386592    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.388285 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.388265    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-kube-api-access-2b8m4" pod "07d433e1-a2e5-46bf-9dbf-e8e946980a1b" (UID: "07d433e1-a2e5-46bf-9dbf-e8e946980a1b"). InnerVolumeSpecName "kube-api-access-2b8m4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:08.388544 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.388525    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tls-certs" pod "07d433e1-a2e5-46bf-9dbf-e8e946980a1b" (UID: "07d433e1-a2e5-46bf-9dbf-e8e946980a1b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:08.388602 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.388529    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-dshm" pod "07d433e1-a2e5-46bf-9dbf-e8e946980a1b" (UID: "07d433e1-a2e5-46bf-9dbf-e8e946980a1b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.487811 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.487766    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-home\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-home\") pod \"d2f49ef7-4390-4190-badf-1b89f8505c04\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") "
Oct 30 14:13:08.487990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.487882    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2f49ef7-4390-4190-badf-1b89f8505c04-tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f49ef7-4390-4190-badf-1b89f8505c04-tls-certs\") pod \"d2f49ef7-4390-4190-badf-1b89f8505c04\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") "
Oct 30 14:13:08.487990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.487969    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2f49ef7-4390-4190-badf-1b89f8505c04-kube-api-access-b5zdw\" (UniqueName: \"kubernetes.io/projected/d2f49ef7-4390-4190-badf-1b89f8505c04-kube-api-access-b5zdw\") pod \"d2f49ef7-4390-4190-badf-1b89f8505c04\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") "
Oct 30 14:13:08.488118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488016    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-tmp-dir\") pod \"d2f49ef7-4390-4190-badf-1b89f8505c04\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") "
Oct 30 14:13:08.488118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488048    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-model-cache\") pod \"d2f49ef7-4390-4190-badf-1b89f8505c04\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") "
Oct 30 14:13:08.488118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488074    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-home" pod "d2f49ef7-4390-4190-badf-1b89f8505c04" (UID: "d2f49ef7-4390-4190-badf-1b89f8505c04"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.488118 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488096    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-dshm\") pod \"d2f49ef7-4390-4190-badf-1b89f8505c04\" (UID: \"d2f49ef7-4390-4190-badf-1b89f8505c04\") "
Oct 30 14:13:08.488352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488309    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-model-cache" pod "d2f49ef7-4390-4190-badf-1b89f8505c04" (UID: "d2f49ef7-4390-4190-badf-1b89f8505c04"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.488456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488429    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.488456 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488455    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.488681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488468    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.488681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488480    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.488681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488493    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b8m4\" (UniqueName: \"kubernetes.io/projected/07d433e1-a2e5-46bf-9dbf-e8e946980a1b-kube-api-access-2b8m4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.488681 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.488254    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-tmp-dir" pod "d2f49ef7-4390-4190-badf-1b89f8505c04" (UID: "d2f49ef7-4390-4190-badf-1b89f8505c04"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.490225 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.490200    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2f49ef7-4390-4190-badf-1b89f8505c04-tls-certs" pod "d2f49ef7-4390-4190-badf-1b89f8505c04" (UID: "d2f49ef7-4390-4190-badf-1b89f8505c04"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:08.490345 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.490240    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2f49ef7-4390-4190-badf-1b89f8505c04-kube-api-access-b5zdw" pod "d2f49ef7-4390-4190-badf-1b89f8505c04" (UID: "d2f49ef7-4390-4190-badf-1b89f8505c04"). InnerVolumeSpecName "kube-api-access-b5zdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:08.490345 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.490285    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-dshm" pod "d2f49ef7-4390-4190-badf-1b89f8505c04" (UID: "d2f49ef7-4390-4190-badf-1b89f8505c04"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:08.590109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.590017    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d2f49ef7-4390-4190-badf-1b89f8505c04-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.590109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.590056    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5zdw\" (UniqueName: \"kubernetes.io/projected/d2f49ef7-4390-4190-badf-1b89f8505c04-kube-api-access-b5zdw\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.590109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.590065    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:08.590109 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:08.590073    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d2f49ef7-4390-4190-badf-1b89f8505c04-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:09.172562 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.172526    4291 generic.go:356] "Generic (PLEG): container finished" podID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerID="9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23" exitCode=0
Oct 30 14:13:09.173044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.172574    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"
Oct 30 14:13:09.173044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.172617    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" event={"ID":"d2f49ef7-4390-4190-badf-1b89f8505c04","Type":"ContainerDied","Data":"9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23"}
Oct 30 14:13:09.173044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.172687    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl" event={"ID":"d2f49ef7-4390-4190-badf-1b89f8505c04","Type":"ContainerDied","Data":"ed9a7e1bf59867436f62a54bfaef7633909d823fff541fdb9da3366a81e6c768"}
Oct 30 14:13:09.173044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.172712    4291 scope.go:122] "RemoveContainer" containerID="9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23"
Oct 30 14:13:09.174407 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.174365    4291 generic.go:356] "Generic (PLEG): container finished" podID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerID="289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723" exitCode=0
Oct 30 14:13:09.174407 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.174392    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" event={"ID":"07d433e1-a2e5-46bf-9dbf-e8e946980a1b","Type":"ContainerDied","Data":"289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723"}
Oct 30 14:13:09.174551 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.174414    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"
Oct 30 14:13:09.174551 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.174426    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh" event={"ID":"07d433e1-a2e5-46bf-9dbf-e8e946980a1b","Type":"ContainerDied","Data":"52e834e094f76421132a452c919a540ee80bd2dd186a5af573e212a724c1363d"}
Oct 30 14:13:09.184251 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.184120    4291 scope.go:122] "RemoveContainer" containerID="9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23"
Oct 30 14:13:09.184524 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:13:09.184501    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23\": container with ID starting with 9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23 not found: ID does not exist" containerID="9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23"
Oct 30 14:13:09.184625 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.184532    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23"} err="failed to get container status \"9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23\": rpc error: code = NotFound desc = could not find container \"9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23\": container with ID starting with 9f30d9df5cd9b8a8ed3ddae551d88e3e6e50abae56753f311d6c549e353bad23 not found: ID does not exist"
Oct 30 14:13:09.184625 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.184550    4291 scope.go:122] "RemoveContainer" containerID="289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723"
Oct 30 14:13:09.193322 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.193303    4291 scope.go:122] "RemoveContainer" containerID="289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723"
Oct 30 14:13:09.193579 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:13:09.193558    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723\": container with ID starting with 289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723 not found: ID does not exist" containerID="289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723"
Oct 30 14:13:09.193652 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.193585    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723"} err="failed to get container status \"289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723\": rpc error: code = NotFound desc = could not find container \"289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723\": container with ID starting with 289d293701059fae275c96adbbc06b95c43c8275f3fcf4f42683d2ae2629a723 not found: ID does not exist"
Oct 30 14:13:09.204999 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.204966    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"]
Oct 30 14:13:09.223076 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.222943    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-22af91-lmdkd"]
Oct 30 14:13:09.232469 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.232431    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"]
Oct 30 14:13:09.235267 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.235238    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/distinct-2-w1-52af49-kserve-55bd98c68-9q2jh"]
Oct 30 14:13:09.264027 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.263994    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"]
Oct 30 14:13:09.269729 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.269707    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-52af49/simulator-w1-52af49-kserve-64b7746d65-q65rl"]
Oct 30 14:13:09.877295 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.877254    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" path="/var/lib/kubelet/pods/07d433e1-a2e5-46bf-9dbf-e8e946980a1b/volumes"
Oct 30 14:13:09.877807 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.877785    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" path="/var/lib/kubelet/pods/d2f49ef7-4390-4190-badf-1b89f8505c04/volumes"
Oct 30 14:13:09.878228 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:09.878208    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae" path="/var/lib/kubelet/pods/f91a4e75-a0f0-4cfd-9f3d-0f8c3dc046ae/volumes"
Oct 30 14:13:11.714276 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:11.714237    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:13:14.757928 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:14.757898    4291 scope.go:122] "RemoveContainer" containerID="01df92db71371080e2f434d9a6ce5fd48a5d5aa240b8ea61e9985167c5cb9c9f"
Oct 30 14:13:14.767603 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:14.767578    4291 scope.go:122] "RemoveContainer" containerID="f94d84ad5546c6bb3f4a93dd604ebc4d5b5030dde1a22323f778cf6cf3ce1967"
Oct 30 14:13:14.776141 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:14.776123    4291 scope.go:122] "RemoveContainer" containerID="55bbf5b6bda6393808fd4698664f749c89060dd114ef49b49de00530641a1191"
Oct 30 14:13:14.784628 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:14.784601    4291 scope.go:122] "RemoveContainer" containerID="4ef22b89322654ffd28ef638a957c37bdce9465d6f656e9d30e7ef93faaea5ea"
Oct 30 14:13:17.927667 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.927605    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"]
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928018    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main"
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928031    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main"
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928043    4291 cpu_manager.go:414] "containerMap: removing container" podUID="7aa44683-5f3b-4df9-94aa-08a7597b4a83" containerName="revoke-keys"
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928048    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aa44683-5f3b-4df9-94aa-08a7597b4a83" containerName="revoke-keys"
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928056    4291 cpu_manager.go:414] "containerMap: removing container" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main"
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928060    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main"
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928067    4291 cpu_manager.go:414] "containerMap: removing container" podUID="c9c6ed43-a1f8-48b7-8b92-86cd31b7b607" containerName="revoke-keys"
Oct 30 14:13:17.928072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928071    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9c6ed43-a1f8-48b7-8b92-86cd31b7b607" containerName="revoke-keys"
Oct 30 14:13:17.928350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928080    4291 cpu_manager.go:414] "containerMap: removing container" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy"
Oct 30 14:13:17.928350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928086    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy"
Oct 30 14:13:17.928350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928137    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2f49ef7-4390-4190-badf-1b89f8505c04" containerName="main"
Oct 30 14:13:17.928350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928145    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aa44683-5f3b-4df9-94aa-08a7597b4a83" containerName="revoke-keys"
Oct 30 14:13:17.928350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928151    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="07d433e1-a2e5-46bf-9dbf-e8e946980a1b" containerName="main"
Oct 30 14:13:17.928350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928156    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9c6ed43-a1f8-48b7-8b92-86cd31b7b607" containerName="revoke-keys"
Oct 30 14:13:17.928350 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.928162    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="de7d1006-839a-487b-89d2-5155bac0c5f1" containerName="istio-proxy"
Oct 30 14:13:17.934779 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.934759    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:17.949674 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.949645    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"]
Oct 30 14:13:17.958827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.958800    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:13:17.958827 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.958823    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:13:17.961812 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.961795    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:13:17.961990 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.961978    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:13:17.993377 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.993343    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn5l9\" (UniqueName: \"kubernetes.io/projected/d6a82d95-29ae-4e79-b3f1-0a982026e401-kube-api-access-wn5l9\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:17.993529 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.993485    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:17.993603 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:17.993549    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.094402 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.094343    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.094598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.094434    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.094598 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.094546    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn5l9\" (UniqueName: \"kubernetes.io/projected/d6a82d95-29ae-4e79-b3f1-0a982026e401-kube-api-access-wn5l9\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.095072 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.095050    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.096833 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.096813    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.113982 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.113951    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn5l9\" (UniqueName: \"kubernetes.io/projected/d6a82d95-29ae-4e79-b3f1-0a982026e401-kube-api-access-wn5l9\") pod \"maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.247565 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.247471    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:18.547122 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:18.547099    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"]
Oct 30 14:13:18.549110 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:13:18.549081    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a82d95_29ae_4e79_b3f1_0a982026e401.slice/crio-42ed45c8760c223f38d84592d7f0267b9f98b0b051971e21ddfedd37c2b66797 WatchSource:0}: Error finding container 42ed45c8760c223f38d84592d7f0267b9f98b0b051971e21ddfedd37c2b66797: Status 404 returned error can't find the container with id 42ed45c8760c223f38d84592d7f0267b9f98b0b051971e21ddfedd37c2b66797
Oct 30 14:13:19.241168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:19.241088    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v" event={"ID":"d6a82d95-29ae-4e79-b3f1-0a982026e401","Type":"ContainerStarted","Data":"a0e6b63b5b8d209746d90afc090471b885c90a48de859a22c71093ac84df22af"}
Oct 30 14:13:19.241168 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:19.241143    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v" event={"ID":"d6a82d95-29ae-4e79-b3f1-0a982026e401","Type":"ContainerStarted","Data":"42ed45c8760c223f38d84592d7f0267b9f98b0b051971e21ddfedd37c2b66797"}
Oct 30 14:13:19.283268 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:19.283217    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v" podStartSLOduration=2.283202228 podStartE2EDuration="2.283202228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 14:13:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:13:19.281716741 +0000 UTC m=+4209.541954979" watchObservedRunningTime="2026-10-30 14:13:19.283202228 +0000 UTC m=+4209.543440480"
Oct 30 14:13:19.976265 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:19.976229    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"]
Oct 30 14:13:19.987532 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:19.987504    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-52af49-c8gkl"]
Oct 30 14:13:20.249052 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:20.248958    4291 generic.go:356] "Generic (PLEG): container finished" podID="d6a82d95-29ae-4e79-b3f1-0a982026e401" containerID="a0e6b63b5b8d209746d90afc090471b885c90a48de859a22c71093ac84df22af" exitCode=0
Oct 30 14:13:20.249501 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:20.249040    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v" event={"ID":"d6a82d95-29ae-4e79-b3f1-0a982026e401","Type":"ContainerDied","Data":"a0e6b63b5b8d209746d90afc090471b885c90a48de859a22c71093ac84df22af"}
Oct 30 14:13:21.394384 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.394360    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:21.529947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.529866    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a82d95-29ae-4e79-b3f1-0a982026e401-kube-api-access-wn5l9\" (UniqueName: \"kubernetes.io/projected/d6a82d95-29ae-4e79-b3f1-0a982026e401-kube-api-access-wn5l9\") pod \"d6a82d95-29ae-4e79-b3f1-0a982026e401\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") "
Oct 30 14:13:21.529947 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.529906    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-openshift-ca\") pod \"d6a82d95-29ae-4e79-b3f1-0a982026e401\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") "
Oct 30 14:13:21.530145 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.530116    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-xks-ca\") pod \"d6a82d95-29ae-4e79-b3f1-0a982026e401\" (UID: \"d6a82d95-29ae-4e79-b3f1-0a982026e401\") "
Oct 30 14:13:21.530315 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.530293    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-openshift-ca" pod "d6a82d95-29ae-4e79-b3f1-0a982026e401" (UID: "d6a82d95-29ae-4e79-b3f1-0a982026e401"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:21.530479 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.530465    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:21.532137 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.532108    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-xks-ca" pod "d6a82d95-29ae-4e79-b3f1-0a982026e401" (UID: "d6a82d95-29ae-4e79-b3f1-0a982026e401"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:21.532259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.532219    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a82d95-29ae-4e79-b3f1-0a982026e401-kube-api-access-wn5l9" pod "d6a82d95-29ae-4e79-b3f1-0a982026e401" (UID: "d6a82d95-29ae-4e79-b3f1-0a982026e401"). InnerVolumeSpecName "kube-api-access-wn5l9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:21.632021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.631986    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d6a82d95-29ae-4e79-b3f1-0a982026e401-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:21.632021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.632019    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn5l9\" (UniqueName: \"kubernetes.io/projected/d6a82d95-29ae-4e79-b3f1-0a982026e401-kube-api-access-wn5l9\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:21.878811 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:21.878779    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="716d3083-3b76-4af3-8a0c-49b876f78c6f" path="/var/lib/kubelet/pods/716d3083-3b76-4af3-8a0c-49b876f78c6f/volumes"
Oct 30 14:13:22.263022 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:22.262943    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"
Oct 30 14:13:22.263184 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:22.262942    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v" event={"ID":"d6a82d95-29ae-4e79-b3f1-0a982026e401","Type":"ContainerDied","Data":"42ed45c8760c223f38d84592d7f0267b9f98b0b051971e21ddfedd37c2b66797"}
Oct 30 14:13:22.263184 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:22.263075    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ed45c8760c223f38d84592d7f0267b9f98b0b051971e21ddfedd37c2b66797"
Oct 30 14:13:22.714594 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:22.714562    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"]
Oct 30 14:13:22.757196 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:22.757156    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:13:29.769745 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.769712    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"]
Oct 30 14:13:29.770157 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.770067    4291 cpu_manager.go:414] "containerMap: removing container" podUID="d6a82d95-29ae-4e79-b3f1-0a982026e401" containerName="revoke-keys"
Oct 30 14:13:29.770157 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.770078    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a82d95-29ae-4e79-b3f1-0a982026e401" containerName="revoke-keys"
Oct 30 14:13:29.770157 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.770154    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a82d95-29ae-4e79-b3f1-0a982026e401" containerName="revoke-keys"
Oct 30 14:13:29.774579 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.774561    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.788385 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.788355    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 14:13:29.791389 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.791365    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"]
Oct 30 14:13:29.799865 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.799837    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2j6ft\""
Oct 30 14:13:29.800039 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.799906    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 14:13:29.800370 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.800341    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 14:13:29.837063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.837010    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.837063 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.837046    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/9c985b1f-7127-4004-b694-acf58e31e2d7-kube-api-access-mk8tf\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.837352 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.837078    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.938076 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.938034    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.938274 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.938084    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/9c985b1f-7127-4004-b694-acf58e31e2d7-kube-api-access-mk8tf\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.938274 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.938158    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.938846 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.938825    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.940457 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.940438    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:29.956937 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:29.956910    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/9c985b1f-7127-4004-b694-acf58e31e2d7-kube-api-access-mk8tf\") pod \"maas-api-revoke-sub-416bf4c2-k4r5g\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:30.087585 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:30.087506    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:30.392304 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:30.392263    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"]
Oct 30 14:13:30.394690 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:13:30.394661    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c985b1f_7127_4004_b694_acf58e31e2d7.slice/crio-bbe78e9be268012eb7aa5edcf8bdeaa070632b4094c32804fda31149640789be WatchSource:0}: Error finding container bbe78e9be268012eb7aa5edcf8bdeaa070632b4094c32804fda31149640789be: Status 404 returned error can't find the container with id bbe78e9be268012eb7aa5edcf8bdeaa070632b4094c32804fda31149640789be
Oct 30 14:13:31.318985 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:31.318945    4291 generic.go:356] "Generic (PLEG): container finished" podID="9c985b1f-7127-4004-b694-acf58e31e2d7" containerID="24a97b66425ca0bbdc353b8592eb3661c17f1719d7e41c755feae632d6d87e4e" exitCode=0
Oct 30 14:13:31.319378 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:31.319005    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g" event={"ID":"9c985b1f-7127-4004-b694-acf58e31e2d7","Type":"ContainerDied","Data":"24a97b66425ca0bbdc353b8592eb3661c17f1719d7e41c755feae632d6d87e4e"}
Oct 30 14:13:31.319378 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:31.319051    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g" event={"ID":"9c985b1f-7127-4004-b694-acf58e31e2d7","Type":"ContainerStarted","Data":"bbe78e9be268012eb7aa5edcf8bdeaa070632b4094c32804fda31149640789be"}
Oct 30 14:13:32.466903 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.466873    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:32.563138 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.563096    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-openshift-ca\") pod \"9c985b1f-7127-4004-b694-acf58e31e2d7\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") "
Oct 30 14:13:32.563326 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.563149    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c985b1f-7127-4004-b694-acf58e31e2d7-kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/9c985b1f-7127-4004-b694-acf58e31e2d7-kube-api-access-mk8tf\") pod \"9c985b1f-7127-4004-b694-acf58e31e2d7\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") "
Oct 30 14:13:32.563326 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.563268    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-xks-ca\") pod \"9c985b1f-7127-4004-b694-acf58e31e2d7\" (UID: \"9c985b1f-7127-4004-b694-acf58e31e2d7\") "
Oct 30 14:13:32.563582 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.563547    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-openshift-ca" pod "9c985b1f-7127-4004-b694-acf58e31e2d7" (UID: "9c985b1f-7127-4004-b694-acf58e31e2d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:32.565386 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.565359    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-xks-ca" pod "9c985b1f-7127-4004-b694-acf58e31e2d7" (UID: "9c985b1f-7127-4004-b694-acf58e31e2d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:32.565463 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.565402    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c985b1f-7127-4004-b694-acf58e31e2d7-kube-api-access-mk8tf" pod "9c985b1f-7127-4004-b694-acf58e31e2d7" (UID: "9c985b1f-7127-4004-b694-acf58e31e2d7"). InnerVolumeSpecName "kube-api-access-mk8tf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:32.664213 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.664178    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-xks-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.664213 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.664212    4291 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c985b1f-7127-4004-b694-acf58e31e2d7-maas-api-openshift-ca\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:32.664437 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:32.664222    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mk8tf\" (UniqueName: \"kubernetes.io/projected/9c985b1f-7127-4004-b694-acf58e31e2d7-kube-api-access-mk8tf\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:33.332954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:33.332915    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g" event={"ID":"9c985b1f-7127-4004-b694-acf58e31e2d7","Type":"ContainerDied","Data":"bbe78e9be268012eb7aa5edcf8bdeaa070632b4094c32804fda31149640789be"}
Oct 30 14:13:33.332954 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:33.332947    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-416bf4c2-k4r5g"
Oct 30 14:13:33.333174 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:33.332971    4291 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe78e9be268012eb7aa5edcf8bdeaa070632b4094c32804fda31149640789be"
Oct 30 14:13:37.734361 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:37.734293    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" containerID="cri-o://476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950" gracePeriod=45
Oct 30 14:13:37.874053 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:37.874015    4291 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": dial tcp 10.133.0.40:8000: connect: connection refused"
Oct 30 14:13:37.988458 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:37.988396    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 14:13:38.018182 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.018086    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a92de03-0008-4266-b100-7ff598142395-kube-api-access-6xmk4\" (UniqueName: \"kubernetes.io/projected/0a92de03-0008-4266-b100-7ff598142395-kube-api-access-6xmk4\") pod \"0a92de03-0008-4266-b100-7ff598142395\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") "
Oct 30 14:13:38.018182 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.018161    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-home\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-home\") pod \"0a92de03-0008-4266-b100-7ff598142395\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") "
Oct 30 14:13:38.018427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.018196    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-tmp-dir\") pod \"0a92de03-0008-4266-b100-7ff598142395\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") "
Oct 30 14:13:38.018427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.018301    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-model-cache\") pod \"0a92de03-0008-4266-b100-7ff598142395\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") "
Oct 30 14:13:38.018427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.018346    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a92de03-0008-4266-b100-7ff598142395-tls-certs\" (UniqueName: \"kubernetes.io/secret/0a92de03-0008-4266-b100-7ff598142395-tls-certs\") pod \"0a92de03-0008-4266-b100-7ff598142395\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") "
Oct 30 14:13:38.018427 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.018375    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-dshm\") pod \"0a92de03-0008-4266-b100-7ff598142395\" (UID: \"0a92de03-0008-4266-b100-7ff598142395\") "
Oct 30 14:13:38.019371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.018982    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-tmp-dir" pod "0a92de03-0008-4266-b100-7ff598142395" (UID: "0a92de03-0008-4266-b100-7ff598142395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.019371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.019274    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-model-cache" pod "0a92de03-0008-4266-b100-7ff598142395" (UID: "0a92de03-0008-4266-b100-7ff598142395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.019371 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.019334    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-home" pod "0a92de03-0008-4266-b100-7ff598142395" (UID: "0a92de03-0008-4266-b100-7ff598142395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.021088 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.021054    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a92de03-0008-4266-b100-7ff598142395-kube-api-access-6xmk4" pod "0a92de03-0008-4266-b100-7ff598142395" (UID: "0a92de03-0008-4266-b100-7ff598142395"). InnerVolumeSpecName "kube-api-access-6xmk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:38.021300 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.021285    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a92de03-0008-4266-b100-7ff598142395-tls-certs" pod "0a92de03-0008-4266-b100-7ff598142395" (UID: "0a92de03-0008-4266-b100-7ff598142395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 14:13:38.021378 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.021317    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-dshm" pod "0a92de03-0008-4266-b100-7ff598142395" (UID: "0a92de03-0008-4266-b100-7ff598142395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:38.120019 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.119982    4291 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-model-cache\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.120019 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.120013    4291 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0a92de03-0008-4266-b100-7ff598142395-tls-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.120019 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.120022    4291 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-dshm\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.120249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.120032    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xmk4\" (UniqueName: \"kubernetes.io/projected/0a92de03-0008-4266-b100-7ff598142395-kube-api-access-6xmk4\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.120249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.120041    4291 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-home\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.120249 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.120071    4291 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0a92de03-0008-4266-b100-7ff598142395-tmp-dir\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:38.210854 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:13:38.210816    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:13:38.363240 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.363204    4291 generic.go:356] "Generic (PLEG): container finished" podID="0a92de03-0008-4266-b100-7ff598142395" containerID="476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950" exitCode=0
Oct 30 14:13:38.363429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.363256    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"
Oct 30 14:13:38.363429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.363295    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" event={"ID":"0a92de03-0008-4266-b100-7ff598142395","Type":"ContainerDied","Data":"476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950"}
Oct 30 14:13:38.363429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.363343    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh" event={"ID":"0a92de03-0008-4266-b100-7ff598142395","Type":"ContainerDied","Data":"f754a2e313aa41835b3e040912ecda7cb938beb93e4263b6074bd1d04221ecc0"}
Oct 30 14:13:38.363429 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.363363    4291 scope.go:122] "RemoveContainer" containerID="476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950"
Oct 30 14:13:38.373158 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.373136    4291 scope.go:122] "RemoveContainer" containerID="476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950"
Oct 30 14:13:38.373444 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:13:38.373422    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950\": container with ID starting with 476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950 not found: ID does not exist" containerID="476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950"
Oct 30 14:13:38.373494 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.373455    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950"} err="failed to get container status \"476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950\": rpc error: code = NotFound desc = could not find container \"476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950\": container with ID starting with 476a8f165049cfd06df764583c6856ddb1f4630a4dbc6f458cd89f6caf2e9950 not found: ID does not exist"
Oct 30 14:13:38.414585 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.414553    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"]
Oct 30 14:13:38.417783 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:38.417762    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-ec41e5/simulator-w2-ec41e5-kserve-66f878ccd6-l8znh"]
Oct 30 14:13:39.878186 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:39.878148    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a92de03-0008-4266-b100-7ff598142395" path="/var/lib/kubelet/pods/0a92de03-0008-4266-b100-7ff598142395/volumes"
Oct 30 14:13:42.789924 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:42.789880    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:13:45.465562 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:45.465358    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"]
Oct 30 14:13:45.476345 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:45.476181    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-ec41e5-crk4v"]
Oct 30 14:13:45.878294 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:45.878255    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a82d95-29ae-4e79-b3f1-0a982026e401" path="/var/lib/kubelet/pods/d6a82d95-29ae-4e79-b3f1-0a982026e401/volumes"
Oct 30 14:13:52.946459 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:52.946418    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-xfsxp"]
Oct 30 14:13:53.191050 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:53.191015    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"]
Oct 30 14:13:53.191232 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:53.191213    4291 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy" containerID="cri-o://90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208" gracePeriod=30
Oct 30 14:13:54.938321 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:54.938288    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qcjns_1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c/kuadrant-console-plugin/0.log"
Oct 30 14:13:55.626661 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:55.626606    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xfsxp_46a07241-a5d6-4177-899b-a851d4760824/limitador/0.log"
Oct 30 14:13:56.852666 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:56.852610    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:13:57.269415 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:57.269331    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:13:58.440802 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.440775    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 14:13:58.497516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.497427    4291 generic.go:356] "Generic (PLEG): container finished" podID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerID="90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208" exitCode=0
Oct 30 14:13:58.497516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.497476    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" event={"ID":"f257e951-f407-4d7f-baaf-24f7327f21d0","Type":"ContainerDied","Data":"90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208"}
Oct 30 14:13:58.497516 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.497487    4291 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"
Oct 30 14:13:58.497837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.497532    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p" event={"ID":"f257e951-f407-4d7f-baaf-24f7327f21d0","Type":"ContainerDied","Data":"6faa5ff41b13bea53f81ba3111abba4e19636608966ddf6918b5e6ca4ab3f5ba"}
Oct 30 14:13:58.497837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.497566    4291 scope.go:122] "RemoveContainer" containerID="90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208"
Oct 30 14:13:58.507413 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.507391    4291 scope.go:122] "RemoveContainer" containerID="90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208"
Oct 30 14:13:58.507738 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:13:58.507712    4291 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208\": container with ID starting with 90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208 not found: ID does not exist" containerID="90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208"
Oct 30 14:13:58.507842 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.507742    4291 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208"} err="failed to get container status \"90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208\": rpc error: code = NotFound desc = could not find container \"90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208\": container with ID starting with 90f0b97d620d8a3bd1548f7ac343fea4253fdf50c0864e3cd2dd56141f605208 not found: ID does not exist"
Oct 30 14:13:58.519981 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.519954    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-credential-socket\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520103 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520008    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-token\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-token\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520103 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520043    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-podinfo\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520103 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520074    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f257e951-f407-4d7f-baaf-24f7327f21d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f257e951-f407-4d7f-baaf-24f7327f21d0-istiod-ca-cert\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520288 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520150    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-certs\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520288 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520185    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-kube-api-access-ggrkv\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-kube-api-access-ggrkv\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520288 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520196    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-credential-socket" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:58.520288 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520244    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-envoy\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520533 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520305    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-data\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520533 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520334    4291 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-socket\") pod \"f257e951-f407-4d7f-baaf-24f7327f21d0\" (UID: \"f257e951-f407-4d7f-baaf-24f7327f21d0\") "
Oct 30 14:13:58.520533 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520463    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-certs" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:58.520730 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520544    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f257e951-f407-4d7f-baaf-24f7327f21d0-istiod-ca-cert" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 14:13:58.520799 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520717    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-socket" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:58.520799 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520781    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-data" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:58.520911 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520872    4291 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.520911 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520892    4291 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-credential-socket\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.520911 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520907    4291 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f257e951-f407-4d7f-baaf-24f7327f21d0-istiod-ca-cert\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.521086 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.520924    4291 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-workload-certs\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.522374 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.522351    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-token" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:58.522474 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.522424    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-kube-api-access-ggrkv" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "kube-api-access-ggrkv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 14:13:58.522612 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.522598    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-podinfo" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 14:13:58.522938 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.522914    4291 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-envoy" pod "f257e951-f407-4d7f-baaf-24f7327f21d0" (UID: "f257e951-f407-4d7f-baaf-24f7327f21d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 14:13:58.622172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.622139    4291 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-token\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.622172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.622169    4291 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-podinfo\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.622172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.622180    4291 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggrkv\" (UniqueName: \"kubernetes.io/projected/f257e951-f407-4d7f-baaf-24f7327f21d0-kube-api-access-ggrkv\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.622417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.622190    4291 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-envoy\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.622417 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.622199    4291 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f257e951-f407-4d7f-baaf-24f7327f21d0-istio-data\") on node \"ip-10-0-136-36.ec2.internal\" DevicePath \"\""
Oct 30 14:13:58.843021 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.842978    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"]
Oct 30 14:13:58.854739 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:58.854709    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-ec41e5-openshift-default-fdd8f456c-tg64p"]
Oct 30 14:13:59.877235 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:13:59.877199    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" path="/var/lib/kubelet/pods/f257e951-f407-4d7f-baaf-24f7327f21d0/volumes"
Oct 30 14:14:08.209973 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:14:08.209931    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:14:10.279970 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:10.279936    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:14:10.701708 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:10.701673    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:14:13.191768 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:13.191733    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-2345757d-p4tz6_7bf90d54-de4e-40f3-8995-08c2610e6172/revoke-keys/0.log"
Oct 30 14:14:13.368070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:13.368038    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-308a008f-gf77p_cc6eebc5-4df0-497b-853d-83701f149949/revoke-keys/0.log"
Oct 30 14:14:13.543078 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:13.542991    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-32a081c7-d4pvp_f9fe5005-7453-4b5e-a3c7-d6d866a42de3/revoke-keys/0.log"
Oct 30 14:14:13.738120 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:13.738088    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-416bf4c2-k4r5g_9c985b1f-7127-4004-b694-acf58e31e2d7/revoke-keys/0.log"
Oct 30 14:14:13.930780 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:13.930752    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-5acba7b8-qt8c9_7aa44683-5f3b-4df9-94aa-08a7597b4a83/revoke-keys/0.log"
Oct 30 14:14:14.117807 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.117779    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-741d51c5-p7zg6_09c8df66-a002-4d78-8542-93d09b0a145d/revoke-keys/0.log"
Oct 30 14:14:14.487768 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.487740    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-84fda1aa-bmqbc_11fa077e-79c2-4ffd-9320-168dd602b772/revoke-keys/0.log"
Oct 30 14:14:14.670526 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.670493    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-98c1547d-46rxs_e57a8441-6a8e-4cc3-9831-499ad518296f/revoke-keys/0.log"
Oct 30 14:14:14.852406 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.852376    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-bc477ec3-sjrsv_c9c6ed43-a1f8-48b7-8b92-86cd31b7b607/revoke-keys/0.log"
Oct 30 14:14:14.890056 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.890028    4291 scope.go:122] "RemoveContainer" containerID="03150863e31d0cdc861836a31837608997e187f5b65acb27cb1e0f7b44844b0d"
Oct 30 14:14:14.899508 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.899488    4291 scope.go:122] "RemoveContainer" containerID="d9287841899ccfa405e55bd5ff987959b30f942d7ae8b76f292614cff7c3370c"
Oct 30 14:14:14.907864 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.907843    4291 scope.go:122] "RemoveContainer" containerID="874b6dc2e69f1e56732b22d3d752956c1df59d3e1c40b52cf6dcd12f5fb1b8cd"
Oct 30 14:14:14.916287 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.916270    4291 scope.go:122] "RemoveContainer" containerID="f9921eba83d10af26318c9fa68c20d9e9b15559e429b53c48912f009f5344dbc"
Oct 30 14:14:14.924649 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.924616    4291 scope.go:122] "RemoveContainer" containerID="b6b6490ee41dc3c4ca42356c47844ea5446ea3dfe0838083328f3c60950430a1"
Oct 30 14:14:14.932424 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.932406    4291 scope.go:122] "RemoveContainer" containerID="5591e560bfa66850af185937addde4a18aec937a4ad2158a1c61354d0801448b"
Oct 30 14:14:14.940145 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.940118    4291 scope.go:122] "RemoveContainer" containerID="ab9164de3711c46eb547baba7c72bd7a64f7e71f2f6fd7aa51408aa249a8f9f3"
Oct 30 14:14:14.947789 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:14.947767    4291 scope.go:122] "RemoveContainer" containerID="bb0a14da76166fa9754c8bf5dcc30657c2f420f234f8c8acc68dd0626ce8c065"
Oct 30 14:14:15.029560 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:15.029535    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-da174186-qg2d8_28952843-8735-4ec2-8f0e-3f0186bf503a/revoke-keys/0.log"
Oct 30 14:14:15.225572 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:15.225496    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e1520bde-8jzhd_46198189-2833-4782-9772-193ca26c986c/revoke-keys/0.log"
Oct 30 14:14:15.421324 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:15.421296    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-e8c6d14a-vs4gq_ca90cf69-045e-4f21-b8aa-9781ebb7d67e/revoke-keys/0.log"
Oct 30 14:14:15.604351 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:15.604325    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fa4e7527-vmz45_c8b43b30-c497-4c3a-935e-e5e7d1914313/revoke-keys/0.log"
Oct 30 14:14:15.796742 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:15.796699    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-fa8454ad-4ckqt_d90dd52e-32c1-4e08-9795-05717432a54c/revoke-keys/0.log"
Oct 30 14:14:18.205630 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:18.205600    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qcjns_1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c/kuadrant-console-plugin/0.log"
Oct 30 14:14:18.797193 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:18.797156    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xfsxp_46a07241-a5d6-4177-899b-a851d4760824/limitador/0.log"
Oct 30 14:14:19.926122 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:19.926092    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fbjhg_fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c/discovery/0.log"
Oct 30 14:14:20.151672 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:20.151624    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7998d67549-78bfj_631405e4-1521-4f58-9a86-9d1e55e60e73/kube-auth-proxy/0.log"
Oct 30 14:14:20.535983 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:20.535954    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-b5924_57ce3df4-b78d-4de7-aecc-5364df414c55/istio-proxy/0.log"
Oct 30 14:14:21.127648 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:21.127607    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-zlnl2_da312528-eb75-4d61-89fb-3c2a6ca8c6aa/istio-proxy/0.log"
Oct 30 14:14:21.323481 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:21.323450    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595_029daaee-d17d-4310-b659-df585ef13a0e/istio-proxy/0.log"
Oct 30 14:14:26.032159 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:26.032076    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"]
Oct 30 14:14:26.043762 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:26.043729    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa4e7527-vmz45"]
Oct 30 14:14:26.045647 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:26.045624    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"]
Oct 30 14:14:26.047327 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:26.047303    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-98c1547d-46rxs"]
Oct 30 14:14:27.876444 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:27.876410    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8b43b30-c497-4c3a-935e-e5e7d1914313" path="/var/lib/kubelet/pods/c8b43b30-c497-4c3a-935e-e5e7d1914313/volumes"
Oct 30 14:14:27.876849 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:27.876830    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e57a8441-6a8e-4cc3-9831-499ad518296f" path="/var/lib/kubelet/pods/e57a8441-6a8e-4cc3-9831-499ad518296f/volumes"
Oct 30 14:14:38.210032 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:14:38.209989    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:14:39.829046 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:39.829012    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j4n8s_37b49c3d-d533-4d1f-898b-2c2fa83e2126/global-pull-secret-syncer/0.log"
Oct 30 14:14:41.157313 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:41.157281    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xnlf9_0a2b2f25-b784-40c6-8fcc-aa4fca8ed8e3/konnectivity-agent/0.log"
Oct 30 14:14:41.575930 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:41.575863    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-36.ec2.internal_3da04c9aa773780d1e2c9af45a694726/haproxy/0.log"
Oct 30 14:14:45.031952 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:45.031916    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"]
Oct 30 14:14:45.043446 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:45.043409    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa8454ad-4ckqt"]
Oct 30 14:14:45.876932 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:45.876899    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90dd52e-32c1-4e08-9795-05717432a54c" path="/var/lib/kubelet/pods/d90dd52e-32c1-4e08-9795-05717432a54c/volumes"
Oct 30 14:14:49.156356 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:49.156330    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-qcjns_1c5f67ee-d4e7-4781-908e-2e2e6e0fc88c/kuadrant-console-plugin/0.log"
Oct 30 14:14:50.240503 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:50.240464    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-xfsxp_46a07241-a5d6-4177-899b-a851d4760824/limitador/0.log"
Oct 30 14:14:51.474845 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.474808    4291 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"]
Oct 30 14:14:51.475259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475165    4291 cpu_manager.go:414] "containerMap: removing container" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main"
Oct 30 14:14:51.475259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475176    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main"
Oct 30 14:14:51.475259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475188    4291 cpu_manager.go:414] "containerMap: removing container" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy"
Oct 30 14:14:51.475259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475193    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy"
Oct 30 14:14:51.475259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475211    4291 cpu_manager.go:414] "containerMap: removing container" podUID="9c985b1f-7127-4004-b694-acf58e31e2d7" containerName="revoke-keys"
Oct 30 14:14:51.475259 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475216    4291 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c985b1f-7127-4004-b694-acf58e31e2d7" containerName="revoke-keys"
Oct 30 14:14:51.475500 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475275    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a92de03-0008-4266-b100-7ff598142395" containerName="main"
Oct 30 14:14:51.475500 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475287    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="f257e951-f407-4d7f-baaf-24f7327f21d0" containerName="istio-proxy"
Oct 30 14:14:51.475500 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.475295    4291 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c985b1f-7127-4004-b694-acf58e31e2d7" containerName="revoke-keys"
Oct 30 14:14:51.478366 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.478350    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.495207 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.495180    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"]
Oct 30 14:14:51.504856 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.504831    4291 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fhsqk\"/\"default-dockercfg-mgtrq\""
Oct 30 14:14:51.504978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.504874    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhsqk\"/\"openshift-service-ca.crt\""
Oct 30 14:14:51.504978 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.504926    4291 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fhsqk\"/\"kube-root-ca.crt\""
Oct 30 14:14:51.521504 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.521475    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-podres\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.521659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.521510    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-lib-modules\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.521659 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.521538    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-proc\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.521756 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.521702    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrfjf\" (UniqueName: \"kubernetes.io/projected/516493e0-4211-4a60-b2f4-4f813663d39d-kube-api-access-qrfjf\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.521824 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.521757    4291 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-sys\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.622967 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.622927    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-podres\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.622967 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.622973    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-lib-modules\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.623229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.623012    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-proc\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.623229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.623064    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrfjf\" (UniqueName: \"kubernetes.io/projected/516493e0-4211-4a60-b2f4-4f813663d39d-kube-api-access-qrfjf\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.623229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.623071    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-podres\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.623229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.623143    4291 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-sys\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.623229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.623150    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-lib-modules\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.623229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.623145    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-proc\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.623229 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.623206    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/516493e0-4211-4a60-b2f4-4f813663d39d-sys\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.641757 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.641732    4291 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrfjf\" (UniqueName: \"kubernetes.io/projected/516493e0-4211-4a60-b2f4-4f813663d39d-kube-api-access-qrfjf\") pod \"perf-node-gather-daemonset-xxq9v\" (UID: \"516493e0-4211-4a60-b2f4-4f813663d39d\") " pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:51.791567 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:51.791488    4291 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:52.093880 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:52.093755    4291 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"]
Oct 30 14:14:52.096753 ip-10-0-136-36 kubenswrapper[4291]: W1030 14:14:52.096729    4291 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod516493e0_4211_4a60_b2f4_4f813663d39d.slice/crio-5bcb4cb232879d25ad6bb838d0e9132897c855bbad950a3bb38a64ad006a882c WatchSource:0}: Error finding container 5bcb4cb232879d25ad6bb838d0e9132897c855bbad950a3bb38a64ad006a882c: Status 404 returned error can't find the container with id 5bcb4cb232879d25ad6bb838d0e9132897c855bbad950a3bb38a64ad006a882c
Oct 30 14:14:52.607430 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:52.607402    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:14:52.816585 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:52.816546    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v" event={"ID":"516493e0-4211-4a60-b2f4-4f813663d39d","Type":"ContainerStarted","Data":"04189561fdb0befcd9c6bb9663d554f512760643c2bc11d5756b30f476116710"}
Oct 30 14:14:52.816817 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:52.816605    4291 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v" event={"ID":"516493e0-4211-4a60-b2f4-4f813663d39d","Type":"ContainerStarted","Data":"5bcb4cb232879d25ad6bb838d0e9132897c855bbad950a3bb38a64ad006a882c"}
Oct 30 14:14:52.816817 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:52.816660    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:14:52.853674 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:52.853607    4291 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v" podStartSLOduration=1.8535930980000002 podStartE2EDuration="1.853593098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 14:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 14:14:52.853205015 +0000 UTC m=+4303.113443243" watchObservedRunningTime="2026-10-30 14:14:52.853593098 +0000 UTC m=+4303.113831326"
Oct 30 14:14:53.390859 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:53.390823    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:14:53.571037 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:53.571001    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:14:54.323364 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:54.323323    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdh7p_6df08611-a79d-4a59-9371-29dfa510b9be/node-exporter/0.log"
Oct 30 14:14:54.457380 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:54.457346    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdh7p_6df08611-a79d-4a59-9371-29dfa510b9be/kube-rbac-proxy/0.log"
Oct 30 14:14:54.483687 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:54.483646    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:14:54.587754 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:54.587654    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-pdh7p_6df08611-a79d-4a59-9371-29dfa510b9be/init-textfile/0.log"
Oct 30 14:14:55.544573 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:55.544536    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a418318-e714-4cad-beda-5aca5d648753/prometheus/0.log"
Oct 30 14:14:55.672223 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:55.672190    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a418318-e714-4cad-beda-5aca5d648753/config-reloader/0.log"
Oct 30 14:14:55.800888 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:55.800802    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a418318-e714-4cad-beda-5aca5d648753/thanos-sidecar/0.log"
Oct 30 14:14:55.929422 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:55.929394    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a418318-e714-4cad-beda-5aca5d648753/kube-rbac-proxy-web/0.log"
Oct 30 14:14:56.031172 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:56.031135    4291 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"]
Oct 30 14:14:56.042837 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:56.042804    4291 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2345757d-p4tz6"]
Oct 30 14:14:56.058044 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:56.057972    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a418318-e714-4cad-beda-5aca5d648753/kube-rbac-proxy/0.log"
Oct 30 14:14:56.185861 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:56.185832    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a418318-e714-4cad-beda-5aca5d648753/kube-rbac-proxy-thanos/0.log"
Oct 30 14:14:56.318262 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:56.318188    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_3a418318-e714-4cad-beda-5aca5d648753/init-config-reloader/0.log"
Oct 30 14:14:56.704748 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:56.704714    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-mmlnt_9becd096-71d7-431d-a6f5-efe50f052a16/prometheus-operator-admission-webhook/0.log"
Oct 30 14:14:56.842658 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:56.842604    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:14:57.243824 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.243787    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8487b78549-kbm4t_8b3eb9b8-8fd3-474b-a3b9-2419e7e68704/thanos-query/0.log"
Oct 30 14:14:57.376467 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.376428    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8487b78549-kbm4t_8b3eb9b8-8fd3-474b-a3b9-2419e7e68704/kube-rbac-proxy-web/0.log"
Oct 30 14:14:57.513921 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.513834    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8487b78549-kbm4t_8b3eb9b8-8fd3-474b-a3b9-2419e7e68704/kube-rbac-proxy/0.log"
Oct 30 14:14:57.624928 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.624891    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:14:57.647176 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.647151    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8487b78549-kbm4t_8b3eb9b8-8fd3-474b-a3b9-2419e7e68704/prom-label-proxy/0.log"
Oct 30 14:14:57.773216 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.773136    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8487b78549-kbm4t_8b3eb9b8-8fd3-474b-a3b9-2419e7e68704/kube-rbac-proxy-rules/0.log"
Oct 30 14:14:57.862963 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.862933    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:14:57.876223 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.876196    4291 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bf90d54-de4e-40f3-8995-08c2610e6172" path="/var/lib/kubelet/pods/7bf90d54-de4e-40f3-8995-08c2610e6172/volumes"
Oct 30 14:14:57.899828 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:57.899793    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-8487b78549-kbm4t_8b3eb9b8-8fd3-474b-a3b9-2419e7e68704/kube-rbac-proxy-metrics/0.log"
Oct 30 14:14:58.794139 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:58.794098    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:14:58.836334 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:14:58.836301    4291 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fhsqk/perf-node-gather-daemonset-xxq9v"
Oct 30 14:15:01.491070 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:01.491024    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:15:02.302283 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:02.302246    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:15:02.768687 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:02.768656    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-rlznc_ff82d94e-deb9-496f-9078-348900979efe/manager/0.log"
Oct 30 14:15:02.884453 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:02.884424    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6dd789c8b7-rmscn_c18f5676-dc4b-45af-a3af-af8db22da5b2/download-server/0.log"
Oct 30 14:15:03.800104 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:03.800065    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-9fqwz_8e3b7ae4-458e-46c5-a75a-d6512feff67b/manager/0.log"
Oct 30 14:15:07.984241 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:07.984201    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nsrbd_95ecd77b-60e5-45f0-b79a-6f9d429f0820/dns/0.log"
Oct 30 14:15:08.109027 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:08.108987    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-nsrbd_95ecd77b-60e5-45f0-b79a-6f9d429f0820/kube-rbac-proxy/0.log"
Oct 30 14:15:08.214906 ip-10-0-136-36 kubenswrapper[4291]: E1030 14:15:08.214856    4291 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-m8257" containerName="extractor"
Oct 30 14:15:08.239809 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:08.239744    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5sstj_20cc5462-99f5-410c-97d6-b414507f297a/dns-node-resolver/0.log"
Oct 30 14:15:09.891966 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:09.891915    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7964b8d7bd-cpfkm_eb1517a6-e078-4162-88ca-27a8da3984d1/registry/0.log"
Oct 30 14:15:10.267815 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:10.267703    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gr5nl_7ec2afa3-085d-4e46-8488-60154d80ec91/node-ca/0.log"
Oct 30 14:15:10.351212 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:10.351172    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-fbjhg_fa8bb1b7-a64f-4b3a-9c05-638a5a08d60c/discovery/0.log"
Oct 30 14:15:10.388270 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:10.388227    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7998d67549-78bfj_631405e4-1521-4f58-9a86-9d1e55e60e73/kube-auth-proxy/0.log"
Oct 30 14:15:10.527256 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:10.527168    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-b5924_57ce3df4-b78d-4de7-aecc-5364df414c55/istio-proxy/0.log"
Oct 30 14:15:10.736561 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:10.736530    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-67bf986f99-zlnl2_da312528-eb75-4d61-89fb-3c2a6ca8c6aa/istio-proxy/0.log"
Oct 30 14:15:10.800730 ip-10-0-136-36 kubenswrapper[4291]: I1030 14:15:10.800621    4291 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_openshift-ai-inference-openshift-ai-inference-67dd9d5b47-js595_029daaee-d17d-4310-b659-df585ef13a0e/istio-proxy/0.log"